homelab/kubernetes

Kubernetes Cluster Bootstrap

This directory contains the Kubernetes infrastructure managed by FluxCD on a 3-node Talos Linux cluster running as VMs on Proxmox.

Prerequisites

Install on your workstation:

• talosctl — Talos cluster management
• kubectl — Kubernetes CLI
• flux — FluxCD CLI
• helm — Helm chart management

Environment variables required:

• GITEA_TOKEN — Gitea personal access token (for FluxCD bootstrap)

Architecture

Proxmox (jupiter, saturn, neptune)
└── 3x Talos Linux VMs (150, 151, 152)
    └── Kubernetes (3 control planes)
        ├── Cilium (CNI, installed via Talos inline manifest)
        └── FluxCD (GitOps, manages everything below)
            ├── MetalLB (L2 load balancer)
            ├── cert-manager (TLS certificates)
            ├── ingress-nginx (ingress controller)
            ├── Sealed Secrets (encrypted secrets)
            ├── CloudNativePG (PostgreSQL operator)
            └── Rook-Ceph (distributed storage)

Directory Structure

kubernetes/
├── talos/                          # Talos machine configs (NOT committed - contain secrets)
│   ├── controlplane.yaml           # Generated base config
│   ├── talosconfig                 # Generated client config
│   ├── patches/
│   │   ├── cilium-cni.yaml         # Disables flannel/kube-proxy, embeds Cilium
│   │   ├── k8s-1.yaml              # Per-node: hostname, disk, certSANs, zone
│   │   ├── k8s-2.yaml
│   │   └── k8s-3.yaml
│   └── cilium.yaml                 # Cilium helm template output
├── infrastructure/                 # Cluster infrastructure (FluxCD priority 1)
│   ├── namespaces/                 # All namespace definitions with PodSecurity labels
│   ├── cilium/                     # Cilium HelmRelease (manages upgrades after bootstrap)
│   ├── metallb/                    # MetalLB HelmRelease
│   ├── cert-manager/               # cert-manager HelmRelease
│   ├── ingress-nginx/              # ingress-nginx HelmRelease
│   ├── sealed-secrets/             # Sealed Secrets HelmRelease
│   ├── cloudnative-pg/             # CloudNativePG operator HelmRelease
│   ├── rook-ceph/                  # Rook-Ceph operator HelmRelease
│   ├── post-crds/                  # Resources that depend on CRDs (ClusterIssuers, IPAddressPools)
│   └── rook-ceph-cluster/          # CephCluster and StorageClasses (depends on operator)
├── apps/                           # Application workloads (FluxCD priority 2)
└── clusters/
    └── homelab/                    # FluxCD entrypoint
        ├── flux-system/            # FluxCD components (auto-managed)
        ├── infrastructure.yaml     # Kustomization → infrastructure/
        └── apps.yaml               # Kustomization → apps/

Clean Bootstrap Procedure

Step 1: Create VMs via OpenTofu

VMs are defined in opentofu/vm_150_k8s_1.tf, vm_151_k8s_2.tf, vm_152_k8s_3.tf.

Push changes and trigger the CI pipeline:

woodpecker pipeline create --branch master \
  --var PIPELINE=deploy-infrastructure \
  --var TOFU_APPLY=true \
  gitea_admin/homelab

The Talos disk image is downloaded to local:import/ on each Proxmox node via the CI pipeline's pre-apply step (Proxmox API download-url). VMs boot directly from the imported disk.

Wait for VMs to appear in DNS: k8s-1.srv.renner.uno, k8s-2.srv.renner.uno, k8s-3.srv.renner.uno.

Step 2: Generate Talos Configs

cd kubernetes/talos

talosctl gen config homelab https://k8s-1.srv.renner.uno:6443 \
  --output-dir . \
  --with-docs=false \
  --with-examples=false \
  --talos-version v1.10 \
  --additional-sans k8s-1.srv.renner.uno,k8s-2.srv.renner.uno,k8s-3.srv.renner.uno \
  --force

Important:

• Use --talos-version v1.10 to match the installed Talos image version
• Use DNS names (not IPs) for the endpoint and SANs — IPs change on VM recreation
• --additional-sans must include all node FQDNs for cross-node TLS

Step 3: Apply Talos Configs

Apply the base config + per-node patch + Cilium CNI patch to each node:

talosctl apply-config --insecure \
  -n k8s-1.srv.renner.uno \
  --file controlplane.yaml \
  --config-patch @patches/k8s-1.yaml \
  --config-patch @patches/cilium-cni.yaml

talosctl apply-config --insecure \
  -n k8s-2.srv.renner.uno \
  --file controlplane.yaml \
  --config-patch @patches/k8s-2.yaml \
  --config-patch @patches/cilium-cni.yaml

talosctl apply-config --insecure \
  -n k8s-3.srv.renner.uno \
  --file controlplane.yaml \
  --config-patch @patches/k8s-3.yaml \
  --config-patch @patches/cilium-cni.yaml

The cilium-cni.yaml patch:

• Disables the default flannel CNI (cluster.network.cni.name: none)
• Disables kube-proxy (cluster.proxy.disabled: true)
• Embeds Cilium as an inline manifest (installed during bootstrap)

Step 4: Configure talosctl and Bootstrap

cp talosconfig ~/.talos/config
talosctl config endpoint k8s-1.srv.renner.uno k8s-2.srv.renner.uno k8s-3.srv.renner.uno
talosctl config node k8s-1.srv.renner.uno

# Wait for all nodes to be visible
talosctl get members

# Bootstrap (run once on the first node)
talosctl bootstrap

# Get kubeconfig
talosctl kubeconfig ~/.kube/config --force

Note: The kubeconfig server URL may need to be updated to use an IPv6 address if IPv4 cross-VLAN routing doesn't work:

K8S1_V6=$(getent ahosts k8s-1.srv.renner.uno | grep -E "^2003" | head -1 | awk '{print $1}')
sed -i "s|server:.*|server: https://[$K8S1_V6]:6443|" ~/.kube/config

Verify all nodes are Ready (Cilium provides CNI from bootstrap):

kubectl get nodes

Step 5: Bootstrap FluxCD

export GITEA_TOKEN=<your-gitea-token>

flux bootstrap gitea \
  --hostname=gitea.renner.uno \
  --owner=gitea_admin \
  --repository=homelab \
  --branch=master \
  --path=kubernetes/clusters/homelab \
  --personal \
  --token-auth

Important: Use --token-auth (HTTPS), not SSH. Gitea's SSH server uses the gitea user, but FluxCD hardcodes git@ which Gitea rejects.

Step 6: Verify

FluxCD reconciles all infrastructure automatically. Monitor progress:

# Watch kustomizations
flux get kustomization --all-namespaces

# Watch helm releases
flux get helmrelease --all-namespaces

# Verify Ceph (takes ~5 minutes)
kubectl get cephcluster -n rook-ceph

All kustomizations and helm releases should show Ready: True. Ceph should reach HEALTH_OK with 3 OSDs.

Updating Cilium

Cilium is installed via Talos inline manifest at bootstrap. After bootstrap, FluxCD manages it via the HelmRelease in infrastructure/cilium/. To update:

1. Update the version in kubernetes/talos/cilium.yaml (regenerate with helm template)
2. Update the version in kubernetes/infrastructure/cilium/helmrelease.yaml
3. Commit and push — FluxCD handles the upgrade

For fresh bootstraps, regenerate patches/cilium-cni.yaml with the new version.

Rebuilding from Scratch

To completely rebuild the cluster:

1. Replace VMs via CI:

   woodpecker pipeline create --branch master \
     --var PIPELINE=deploy-infrastructure \
     --var TOFU_APPLY=true \
     --var "TOFU_REPLACE=module.k8s_1.proxmox_virtual_environment_vm.this module.k8s_2.proxmox_virtual_environment_vm.this module.k8s_3.proxmox_virtual_environment_vm.this" \
     gitea_admin/homelab
   

2. Follow Steps 2-6 above

Known Gotchas

• DNS names, not IPs: Always use *.srv.renner.uno DNS names. DHCP IPs change on VM recreation.
• IPv6 for kubectl: Cross-VLAN IPv4 may not reach the K8s API. Use IPv6 in kubeconfig.
• Talos version flag: Use --talos-version v1.10 when generating configs to match the installed image.
• FluxCD token auth: Use --token-auth with GITEA_TOKEN, never SSH bootstrap with Gitea.
• CRD ordering: Infrastructure is split into stages (base → post-crds → ceph-cluster) because FluxCD dry-run fails if CRDs don't exist yet.
• PodSecurity labels: Namespaces metallb-system, rook-ceph, ingress need pod-security.kubernetes.io/enforce: privileged.
• Ceph disk cleanup: If VMs are recreated, old Ceph data on vdb may prevent OSD creation. A fresh VM with a fresh disk avoids this.