Files
homelab/kubernetes/bootstrap.sh
T
claude-botandClaude Opus 4.8 6900271f86
ci/woodpecker/pr/pr-review Pipeline was successful
refactor(woodpecker): dedicated woodpecker KS, retire seed-woodpecker (stage 2)
Restructures woodpecker into its own Flux Kustomization instead of splitting its
app across the catch-all apps KS and its DB across seed-woodpecker.

- Add clusters/homelab/woodpecker.yaml: dedicated KS over apps/woodpecker with
  the infrastructure-cnpg-barman dep + CNPG Cluster healthCheckExprs (moved off
  seed-woodpecker) so the CNPG concerns live here, not in the shared apps KS.
- Remove woodpecker/ from apps/kustomization.yaml (catch-all apps KS no longer
  owns the woodpecker app objects).
- git mv seed/woodpecker/{cluster,objectstore,scheduled-backup,
  hetzner-creds-externalsecret}.yaml + MIGRATION.md -> apps/woodpecker/, and add
  the four DB manifests to apps/woodpecker/kustomization.yaml.
- Delete clusters/homelab/seed-woodpecker.yaml and the now-empty
  kubernetes/seed/woodpecker/.
- bootstrap.sh: add a wait_kustomization woodpecker entry (it used to be covered
  by 'wait apps'); e2e-test.sh: add woodpecker to the KS readiness checks.

Safe dual ownership transfer (nothing pruned): every object carries prune:
disabled from stage 1 (#394) — the app objects via commonAnnotations, the DB
objects per-file — so when the apps KS drops woodpecker/ and seed-woodpecker is
deleted, both GC sweeps skip these live objects and the new woodpecker KS adopts
them in place. No CNPG restart (Cluster is not a Deployment); no app restart
(manifests unchanged, only the owning KS label flips).

Verified: kubectl kustomize apps/woodpecker builds 17 objects (13 app + 4 DB
incl. wp-pg Cluster), all prune-protected; bash -n bootstrap.sh + e2e-test.sh
OK; wp-pg is the sole CNPG cluster in apps ns and hetzner-s3-creds is used only
by wp-pg-store; seed/woodpecker fully removed.

Stage 3 will drop the transient commonAnnotations (one more accepted app roll).

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 06:18:54 +02:00

396 lines
17 KiB
Bash
Executable File
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/usr/bin/env bash
set -euo pipefail
# K8s Cluster Bootstrap — split into phases so CI can export access as early as possible.
#
# Usage:
# ./bootstrap.sh # run all phases locally
# ./bootstrap.sh talos # Talos apply + K8s bootstrap + kubeconfig (writes access-staging/)
# ./bootstrap.sh flux # reads access-staging/, installs flux, waits for infrastructure
# ./bootstrap.sh wait-apps # waits for apps/authentik/paperless to reconcile
#
# Flux git source (break-glass): the in-cluster Gitea that Flux normally pulls
# from is itself Flux-managed, so a cold rebuild cannot pull from it until the
# cluster already exists. Set FLUX_SOURCE=github to bootstrap Flux from the
# external GitHub mirror instead — the only way out of that loop:
# FLUX_SOURCE=github GITHUB_TOKEN=<pat> ./bootstrap.sh flux
# The mirror is kept current server-side by Gitea's native push-mirror. Full
# cold-rebuild procedure: kubernetes/RUNBOOK-cold-rebuild.md.
#
# Requires: talosctl, kubectl, flux, sops age key at ~/.config/sops/age/keys.txt
TALOS_DIR="$(cd "$(dirname "$0")/talos" && pwd)"
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
ACCESS_STAGING="$REPO_ROOT/kubernetes/access-staging"
NODES=(k8s-1 k8s-2 k8s-3)
# No dedicated workers since the k8s-edge VM was retired (2026-07-02) —
# hardware-bound services live on the bare-metal edge-1 Arch host
# (ansible/roles/edge-host).
WORKERS=()
ALL_NODES=("${NODES[@]}" "${WORKERS[@]}")
DOMAIN="srv.renner.uno"
DNS_SERVER="10.2.40.1" # UDM (old PVE DNS VIP 10.2.40.53 decommissioned)
# Cluster API endpoint — Talos VIP (10.2.40.60) elected via etcd, fronted by
# UniFi DNS A record k8s.renner.uno. Defined in talos/patches/k8s-{1,2,3}.yaml.
K8S_API_HOST="k8s.renner.uno"
GITEA_HOST="gitea.renner.uno"
GITEA_OWNER="gitea_admin"
GITEA_REPO="homelab"
# Flux git source: "gitea" (default, in-cluster) or "github" (break-glass mirror
# for a cold rebuild — see the usage note above and RUNBOOK-cold-rebuild.md).
FLUX_SOURCE="${FLUX_SOURCE:-gitea}"
# GitHub break-glass mirror (github.com/konairius/Homelab), kept current by
# Gitea's native push-mirror. Overridable in case the mirror moves.
GITHUB_OWNER="${GITHUB_OWNER:-konairius}"
GITHUB_REPO="${GITHUB_REPO:-Homelab}"
FLUX_PATH="kubernetes/clusters/homelab"
SOPS_KEY="$HOME/.config/sops/age/keys.txt"
PHASE="${1:-all}"
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'
is_tty() { [[ -t 1 ]]; }
step() {
local _ignored=$1
local label="$2"
if is_tty; then
printf "${BLUE}==>${NC} %s\n" "$label"
else
printf "==> %s\n" "$label"
fi
}
fail() { echo -e "\n${RED}FAILED:${NC} $1"; exit 1; }
ok() { echo -e "${GREEN}OK${NC}"; }
resolve_node_ip() {
local fqdn=$1
dig +short "$fqdn" A @"$DNS_SERVER" | head -1
}
phase_runs() {
[[ "$PHASE" == "all" || "$PHASE" == "$1" ]]
}
if phase_runs talos; then
echo -e "${YELLOW}=== Phase: Talos bootstrap ===${NC}"
command -v talosctl >/dev/null || fail "talosctl not found"
command -v kubectl >/dev/null || fail "kubectl not found"
[[ -f "$TALOS_DIR/patches/cluster.yaml" ]] || fail "cluster.yaml patch not found"
[[ -f "$TALOS_DIR/patches/cilium-cni.yaml" ]] || fail "cilium-cni.yaml patch not found"
[[ -f "$TALOS_DIR/patches/registry-mirror.yaml" ]] || fail "registry-mirror.yaml patch not found"
step 0 "Generating Talos configs..."
TALOS_GEN_DIR=$(mktemp -d)
talosctl gen config homelab "https://${K8S_API_HOST}:6443" \
--output "$TALOS_GEN_DIR" \
--force
# Strip the auto-generated HostnameConfig docs (conflicts with machine.network.hostname in per-node patches)
for f in "$TALOS_GEN_DIR"/controlplane.yaml "$TALOS_GEN_DIR"/worker.yaml; do
awk '
BEGIN { doc="" }
/^---[ \t]*$/ {
if (doc !~ /\nkind: HostnameConfig\n/ && doc !~ /^kind: HostnameConfig\n/) printf "%s", doc
doc=$0 "\n"; next
}
{ doc = doc $0 "\n" }
END {
if (doc !~ /\nkind: HostnameConfig\n/ && doc !~ /^kind: HostnameConfig\n/) printf "%s", doc
}
' "$f" > "$f.tmp" && mv "$f.tmp" "$f"
done
mkdir -p ~/.talos
cp "$TALOS_GEN_DIR/talosconfig" ~/.talos/config
step 0 "Applying Talos configs to nodes..."
apply_node_config() {
local node=$1 base_config=$2
local fqdn="${node}.${DOMAIN}"
echo " $fqdn:"
for i in $(seq 1 60); do
node_ip=$(resolve_node_ip "$fqdn")
if [[ -z "$node_ip" ]]; then
echo " attempt $i/60: DNS returned no IP"
sleep 10
continue
fi
if err=$(talosctl apply-config --insecure -n "$node_ip" \
--file "$base_config" \
--config-patch "@$TALOS_DIR/patches/cluster.yaml" \
--config-patch "@$TALOS_DIR/patches/cilium-cni.yaml" \
--config-patch "@$TALOS_DIR/patches/registry-mirror.yaml" \
--config-patch "@$TALOS_DIR/patches/${node}.yaml" 2>&1); then
echo " applied via $node_ip ${GREEN}(OK)${NC}"
return 0
fi
echo " attempt $i/60: $node_ip - ${err//$'\n'/ | }"
sleep 10
done
fail "Timeout waiting for $fqdn"
}
for node in "${NODES[@]}"; do
apply_node_config "$node" "$TALOS_GEN_DIR/controlplane.yaml"
done
for node in "${WORKERS[@]}"; do
apply_node_config "$node" "$TALOS_GEN_DIR/worker.yaml"
done
echo -n "Configuring talosctl... "
talosctl config endpoint "${NODES[@]/%/.${DOMAIN}}"
talosctl config node "${NODES[0]}.${DOMAIN}"
ok
step 0 "Waiting for Talos cluster formation..."
for i in $(seq 1 30); do
members=$(talosctl get members 2>/dev/null || true)
missing=0
for node in "${ALL_NODES[@]}"; do
grep -q "$node" <<<"$members" || missing=1
done
[[ $missing -eq 0 ]] && break
sleep 5
[[ $i -eq 30 ]] && fail "Talos cluster didn't form"
done
echo -n "Bootstrapping Kubernetes... "
talosctl bootstrap 2>/dev/null || true
ok
echo -n "Getting kubeconfig... "
for i in $(seq 1 30); do
if talosctl kubeconfig ~/.kube/config --force 2>/dev/null; then
sed -i "s|server:.*|server: https://${K8S_API_HOST}:6443|" ~/.kube/config
break
fi
sleep 5
[[ $i -eq 30 ]] && fail "Couldn't get kubeconfig"
done
ok
step 0 "Waiting for K8s nodes..."
expected=${#ALL_NODES[@]}
for i in $(seq 1 60); do
ready=$(kubectl get nodes --no-headers 2>/dev/null | grep -c "Ready" || true)
if [[ "$ready" -eq "$expected" ]]; then
break
fi
sleep 5
[[ $i -eq 60 ]] && fail "Not all nodes ready (got $ready/$expected)"
done
# Stash configs for subsequent phases / export-access step
mkdir -p "$ACCESS_STAGING"
cp ~/.talos/config "$ACCESS_STAGING/talosconfig"
cp ~/.kube/config "$ACCESS_STAGING/kubeconfig"
echo -e "${GREEN}Talos phase complete.${NC} Access files in $ACCESS_STAGING/"
fi
# Helper: load staged access files into home dir (for later CI steps)
load_staged_access() {
[[ -f "$ACCESS_STAGING/talosconfig" && -f "$ACCESS_STAGING/kubeconfig" ]] \
|| fail "access-staging/ missing talosconfig or kubeconfig (run talos phase first)"
mkdir -p ~/.talos ~/.kube
cp "$ACCESS_STAGING/talosconfig" ~/.talos/config
cp "$ACCESS_STAGING/kubeconfig" ~/.kube/config
}
if phase_runs flux; then
echo -e "${YELLOW}=== Phase: FluxCD bootstrap ===${NC}"
command -v kubectl >/dev/null || fail "kubectl not found"
command -v flux >/dev/null || fail "flux not found"
[[ -f "$SOPS_KEY" ]] || fail "SOPS age key not found at $SOPS_KEY"
case "$FLUX_SOURCE" in
gitea) [[ -n "${GITEA_TOKEN:-}" ]] || fail "GITEA_TOKEN env var not set" ;;
github) [[ -n "${GITHUB_TOKEN:-}" ]] || fail "GITHUB_TOKEN env var not set (break-glass mirror bootstrap)" ;;
*) fail "Unknown FLUX_SOURCE '$FLUX_SOURCE' (want gitea|github)" ;;
esac
[[ "$PHASE" == "flux" ]] && load_staged_access
echo -n "Creating SOPS age secret... "
kubectl create namespace flux-system --dry-run=client -o yaml | kubectl apply -f - 2>/dev/null
kubectl create secret generic sops-age -n flux-system \
--from-file=age.agekey="$SOPS_KEY" \
--dry-run=client -o yaml | kubectl apply -f - 2>/dev/null
ok
if [[ "$FLUX_SOURCE" == "github" ]]; then
step 0 "Bootstrapping FluxCD from the GitHub break-glass mirror..."
echo -e "${YELLOW}BREAK-GLASS: Flux source is github.com/${GITHUB_OWNER}/${GITHUB_REPO}, NOT in-cluster Gitea.${NC}"
echo -e "${YELLOW}Switch the source back to Gitea once it is serving again (see RUNBOOK-cold-rebuild.md).${NC}"
# GITHUB_TOKEN is read from the environment by `flux bootstrap github`.
flux bootstrap github \
--owner="$GITHUB_OWNER" \
--repository="$GITHUB_REPO" \
--branch=master \
--path="$FLUX_PATH" \
--components-extra=image-reflector-controller,image-automation-controller \
--personal \
--token-auth 2>&1 | tail -5
else
step 0 "Bootstrapping FluxCD from in-cluster Gitea..."
flux bootstrap gitea \
--hostname="$GITEA_HOST" \
--owner="$GITEA_OWNER" \
--repository="$GITEA_REPO" \
--branch=master \
--path="$FLUX_PATH" \
--components-extra=image-reflector-controller,image-automation-controller \
--personal \
--token-auth 2>&1 | tail -5
fi
step 0 "Waiting for infrastructure..."
for i in $(seq 1 60); do
ready=$(flux get kustomization infrastructure --no-header 2>/dev/null | grep -c "True" || true)
if [[ "$ready" -ge 1 ]]; then
break
fi
sleep 10
[[ $i -eq 60 ]] && echo -e "${YELLOW}Warning: infrastructure not ready yet, continuing...${NC}"
done
step 0 "Waiting for Ceph cluster..."
for i in $(seq 1 60); do
health=$(kubectl get cephcluster -n rook-ceph -o jsonpath='{.items[0].status.phase}' 2>/dev/null || true)
if [[ "$health" == "Ready" ]]; then
break
fi
sleep 10
[[ $i -eq 60 ]] && echo -e "${YELLOW}Warning: Ceph not ready yet, continuing...${NC}"
done
step 0 "Waiting for Ceph Object Store..."
for i in $(seq 1 60); do
phase=$(kubectl get cephobjectstore -n rook-ceph migration -o jsonpath='{.status.phase}' 2>/dev/null || true)
if [[ "$phase" == "Ready" ]]; then
break
fi
sleep 10
[[ $i -eq 60 ]] && echo -e "${YELLOW}Warning: Object Store not ready yet, continuing...${NC}"
done
fi
if phase_runs wait-apps; then
echo -e "${YELLOW}=== Phase: Wait for apps ===${NC}"
command -v kubectl >/dev/null || fail "kubectl not found"
command -v flux >/dev/null || fail "flux not found"
[[ "$PHASE" == "wait-apps" ]] && load_staged_access
# Wait for a Flux Kustomization to become Ready=True. Prints a status line
# whenever the Ready/Message pair changes, plus a heartbeat every 60 s so a
# silent-but-progressing reconcile is still visible in CI output. Returns
# 0 on Ready, 1 on timeout (caller decides whether to fail or continue).
wait_kustomization() {
local name="$1" label="$2" max_iters="$3"
local interval=10 elapsed=0 prev_status=""
local timeout_min=$(( max_iters * interval / 60 ))
step 0 "Waiting for ${label} (timeout ${timeout_min}m)..."
for _ in $(seq 1 "$max_iters"); do
local line ready msg status
line=$(flux get kustomization "$name" --no-header 2>/dev/null || true)
# `flux get kustomization --no-header` columns:
# NAME REVISION SUSPENDED READY MESSAGE...
ready=$(echo "$line" | awk '{print $4}')
msg=$(echo "$line" | awk '{for(i=5;i<=NF;i++) printf "%s ",$i}' | sed 's/[[:space:]]*$//')
if [[ "$ready" == "True" ]]; then
printf " [%4ds] Ready ✓\n" "$elapsed"
return 0
fi
status="${ready:-?}: ${msg:-(no status yet)}"
# Print on first iteration, on state change, or as a 60-s heartbeat.
# 240 chars is enough to see the full Flux stalled-resource message.
if [[ "$prev_status" != "$status" ]] || (( elapsed > 0 && elapsed % 60 == 0 )); then
printf " [%4ds] %s\n" "$elapsed" "${status:0:240}"
prev_status="$status"
fi
sleep "$interval"
elapsed=$(( elapsed + interval ))
done
printf " [%4ds] TIMEOUT after %sm — full status follows\n" "$elapsed" "$timeout_min"
# Dump the full untruncated status + Events for the offending resources
# so the post-mortem doesn't need a second cluster trip.
flux get kustomization "$name" 2>&1 | sed 's/^/ /' || true
kubectl get events --sort-by=.lastTimestamp -A 2>/dev/null \
| grep -i -E "warning|error" | tail -20 | sed 's/^/ /' || true
echo -e "${YELLOW}Warning: ${label} not Ready, continuing anyway${NC}"
return 1
}
# Per-app seed KSes (`seed-<app>`) populate each CNPG database and
# each migrating PVC from legacy hosts. Each downstream `<app>` KS
# gates only on its own `seed-<app>`, so a slow seed (immich-library
# rsync at ~26 min) no longer blocks the unrelated apps. The per-app
# waits below absorb both the seed-<app> runtime and the app rollout.
# apps waits on infrastructure-ceph-cluster (Ceph mon/mgr/OSD bring-up
# is genuinely 5–10 min on a fresh OSD-wiped cluster).
wait_kustomization apps "apps" 90
wait_kustomization influxdb "InfluxDB (HR install + token Jobs)" 600
wait_kustomization woodpecker "Woodpecker CI (incl. wp-pg CNPG backend)" 600
wait_kustomization authentik "Authentik (incl. ak-pg seed)" 600
wait_kustomization monitoring "kube-prometheus-stack (Loki + Prom + AM + Grafana)" 180
wait_kustomization music-assistant "Music Assistant (incl. data seed)" 300
wait_kustomization paperless "Paperless (incl. pprl-pg + data seed)" 600
wait_kustomization home-assistant "Home Assistant (HAOS backup → native restore)" 180
wait_kustomization audiobookshelf "Audiobookshelf (incl. ~70 GiB library seed)" 900
wait_kustomization immich "Immich (incl. ~26 min library-seed rsync)" 1800
# Wait for cert-manager to finish issuing every Certificate. Kustomization
# Ready only means the Ingress + Certificate manifests are applied; cert-
# manager then runs an async ACME DNS-01 challenge per Certificate (~30s–2m
# via Cloudflare). e2e's TLS check fails on the ingress-nginx default fake
# cert if any Certificate is still Issuing when the test runs.
wait_certificates() {
local max_iters=90 interval=10 elapsed=0 prev_status=""
local timeout_min=$(( max_iters * interval / 60 ))
step 0 "Waiting for all cert-manager Certificates Ready (timeout ${timeout_min}m)..."
for _ in $(seq 1 "$max_iters"); do
local total ready not_ready_names status
total=$(kubectl get certificate -A --no-headers 2>/dev/null | wc -l | tr -d ' ')
ready=$(kubectl get certificate -A --no-headers 2>/dev/null | awk '$3=="True"' | wc -l | tr -d ' ')
if [[ "$total" -gt 0 && "$total" == "$ready" ]]; then
printf " [%4ds] All %s Certificates Ready ✓\n" "$elapsed" "$total"
return 0
fi
not_ready_names=$(kubectl get certificate -A --no-headers 2>/dev/null \
| awk '$3!="True" {print $1"/"$2}' | head -5 | tr '\n' ' ')
status="${ready:-0}/${total:-?} Ready, waiting: ${not_ready_names:-(none)}"
if [[ "$prev_status" != "$status" ]] || (( elapsed > 0 && elapsed % 60 == 0 )); then
printf " [%4ds] %s\n" "$elapsed" "${status:0:140}"
prev_status="$status"
fi
sleep "$interval"
elapsed=$(( elapsed + interval ))
done
printf " [%4ds] TIMEOUT after %sm\n" "$elapsed" "$timeout_min"
echo -e "${YELLOW}Warning: not all Certificates Ready, continuing anyway${NC}"
kubectl get certificate -A --no-headers 2>/dev/null | awk '$3!="True"' | head -10
return 1
}
wait_certificates
echo ""
echo -e "${YELLOW}=== Verification ===${NC}"
echo -n "Nodes: "
kubectl get nodes --no-headers 2>/dev/null | grep -c Ready | tr -d '\n'
echo "/${#ALL_NODES[@]}"
echo -n "HelmReleases: "
flux get helmrelease --all-namespaces --no-header 2>/dev/null | grep -c "True" | tr -d '\n'
echo " ready"
echo -n "Ceph: "
kubectl get cephcluster -n rook-ceph -o jsonpath='{.items[0].status.ceph.health}' 2>/dev/null || echo "pending"
echo ""
echo -n "Ingress (local): "
curl -sk --connect-timeout 5 --resolve status.renner.uno:443:10.2.99.86 https://status.renner.uno -o /dev/null -w "%{http_code}" 2>/dev/null || echo "N/A"
echo ""
fi