fix/wordpress-php85-php2shell
69
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  claude-botandClaude Opus 5.5
|
c862082d0e |
feat(restore-test): hcloudtest boots production's config on Hetzner, contained, and Flux restores into it
The restore rehearsal: a throwaway three-node cluster on Hetzner Cloud that runs the production tree and restores production's backups into itself. hcloudtest (bootstrap.sh) runs SITE=hcloud on cpx52, and nothing stands in for the home network: - site-hcloud.yaml states the Hetzner site's facts, as site-homelab.yaml does at home: etcd advertised on the node segment 10.90.0.0/16; a cilium-site ConfigMap with tunnel (VXLAN) routing over the node segment and only the pod CIDRs exempt from masquerading (Hetzner networks are layer 3, IPv4-only, and drop packets with a source not assigned to the sending server); a ResolverConfig with k8s-gateway first and a k8s-gateway-site ConfigMap pointing at Hetzner's resolvers. The node range is read from the machine document only, now that the patch holds more than one (run 28). - hcloud-1..3.yaml put a 345GiB OSD partition beside a 60GiB EPHEMERAL on the system disk: the project's volume quota cannot hold three OSD volumes. No Hetzner volume is created for a raw volume on the system disk. - The Talos snapshot is built when missing (a rescue-mode server writes the Image Factory image; removed on failure), matched by version and schematic. This host's inbound firewall replaces earlier runs' without a server ever being left without one. The Hetzner token comes from OpenBao (homelab-backup-verify project). Access files land in ~/.local/state/homelab-hcloudtest instead of access-staging. - Containment: an outbound firewall on the nodes, applied before any config, admits udp and icmp anywhere and tcp to the complement of proxy.renner.uno's addresses, resolved over DoH at run time (cidr_complement, in bash; fails above Hetzner's 500 effective rules; one tcp rule per 100 prefixes, Hetzner's undocumented per-rule limit). The egress check pod (restore-test/egress-check.yaml) runs at the end of hcloudtest and again before the restore test writes any secret. FLUX_SOURCE=restore-test bootstrap.sh flux runs against that cluster only (its nodes must all be hcloud-*, and KUBECONFIG/TALOSCONFIG must not be the operator's): flux install, a per-run read-only deploy key on the GitHub mirror, the restore-test-git Secret, and restore-test/flux-root.yaml on FLUX_BRANCH. flux-root.yaml applies the production tree; its patches are the test's whole deviation: the source rewrite, no Ceph cluster network, Velero ReadOnly, no MariaDB dumps, rotators suspended, Gitea without world egress, netbird-service-sync at zero replicas and netbird-ingress on no node (the test cluster rewrote production NetBird's proxy services in run 28), and backup-verify suspended: it reads the same Hetzner token as hcloudtest and would delete production's monthly verify VM (both label managed-by=backup-verify). The patches are labelled 3a to 3h without gaps. The egress check resolves A and AAAA separately and fails on either error; before, a failed A lookup went unnoticed and the check ran on IPv6 alone. Removing a previous egress-check pod fails with a message. hcloudclean checks for bao like hcloudtest does. site-hcloud.yaml's header says what the site is for now and that Ceph replicates over the pod network there. CNPG archives, backs up and prunes as in production: the flux phase writes cnpg-paths with every archive path in renner-uno-k8s-cnpg-rtest, created there and emptied and deleted by hcloudclean. A test incarnation with a base backup in the production bucket would be the newest there, and the next real recovery would read the test's data. hcloudclean also deletes the test's ssh keys and the restore-test deploy keys. OpenBao's status wait is bounded at 30 minutes: on a fresh cluster the pod waits for Ceph to provision its volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5.5
|
a17c4e8161 |
feat(cnpg): every cluster incarnation archives to a new prefix; recovery reads the newest
A CNPG cluster restored from barman archived back into the location it
recovered from. plugin-barman-cloud's restore job checks the archive target
(plugins[] ObjectStore + serverName) and stops with "Expected empty archive"
when it holds WAL, which after a disaster it always does. It has no
exception for target == recovery source (internal/cnpgi/restore/restore.go,
v0.15). Restore rehearsal run 28 hit it on wp-pg; the other six never got
that far.
The check stays. Instead, archive and source become two ObjectStores whose
paths are chosen per incarnation:
- kubernetes/cnpg-object-stores/: all 14 ObjectStores (<x>-store archives,
<x>-store-source is read by bootstrap.recovery), in a Kustomization of their
own with postBuild.substituteFrom: cnpg-paths. Nothing else lives there:
substitution blanks every ${...}, which is why it is not on the app
Kustomizations (home-assistant's scripts, the Grafana dashboards). The
ConfigMap is not optional; without it the paths would render empty.
woodpecker's objectstore.yaml moves there as wp-pg.yaml.
- bootstrap.sh writes cnpg-paths in the flux phase, before Flux starts, once
per incarnation (an existing one is kept):
CNPG_<X>_SOURCE the prefix among <x>, <x>-g<14 digits> and the store's
cnpg.renner.uno/source-prefixes whose <x>/base/ holds
the newest completed base backup (backup.info
status=DONE; barman uploads it as STARTED when a
backup begins, barman src/barman/cloud.py
coordinate_backup)
CNPG_<X>_ARCHIVE <x>-g<UTC timestamp>, new and so empty
Every ScheduledBackup has immediate: true, so a new incarnation has a base
backup in its own prefix minutes after it starts, and a later recovery
picks it. S3 access is curl --aws-sigv4 with the restore key (already in
bootstrap.sh's secret list), so the CI image needs no new tool; the
OpenBao restore reads the same S3 endpoint and region.
- The six clusters that recovered from their archive store now name
<x>-store-source in externalClusters. serverName stays the cluster name
everywhere. The app Kustomizations depend on cnpg-object-stores, which
depends on infrastructure-cnpg-barman.
Verified against the bucket, read-only, 2026-10-16: the restore key lists
renner-uno-k8s-cnpg; the listing parser on real responses, top level and an
encoded nested prefix; for wp-pg it picks wp-pg-woodpecker (newest
20261016T024000) over wp-pg (20260818T023502); leftover test serverNames
hold no base backups and match no candidate pattern.
Production rollout, one time:
1. Create flux-system/cnpg-paths with the current prefixes as both source and
archive (ak-pg, ha-pg, im-pg, mx-pg, pprl-pg, tr-pg; wp-pg-woodpecker), so
the running clusters keep archiving where they do.
2. The app Kustomizations that give up their ObjectStores (authentik,
home-assistant, immich, matrix, paperless, trmnl, woodpecker) must run
with prune: false (the stack's first PR) when this merges;
cnpg-object-stores adopts the objects. All 14 also carry prune: disabled.
3. Whether changing externalClusters on a live Cluster restarts its pods is
not verified; check on one cluster first.
Older generations are not cleaned up: retention applies only to the current
archive prefix.
RECOVERY.md describes the new layout: where the current prefixes are
(cnpg-paths), the per-incarnation backup paths for the precondition check,
archiving to the new prefix after promotion, and adding a CNPG cluster to a
running cluster (its keys go into the existing cnpg-paths by hand). The
manual restore drill reads <x>-store, the current archive. The app
kustomization comments name cnpg-object-stores instead of
infrastructure-cnpg-barman. write_cnpg_paths checks for yq first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
claude-botandClaude Opus 5.5
|
93cb53130c |
feat(velero): the cluster restores its app volumes itself, Gitea first
One Velero Restore per Schedule (kubernetes/restore/<schedule>/), applied
by Flux through a restore-<schedule> Kustomization
(clusters/homelab/restore.yaml), restoring the schedule's newest completed
backup. Every app Kustomization with backed-up volumes depends on its own
restore Kustomization, so on a fresh cluster no app starts on an empty
volume, and no app waits for another app's data. This reverses the
decision on #200 (option B,
|
||
|
claude-botandClaude Opus 5.5
|
c7b504a3f5 |
fix(bootstrap): k8s-gateway values with either yq; cni-confdir step re-runs
- The k8s-gateway values were read with `yq -y`, a flag only the Python yq has. The vmtest image installs Alpine's `yq`, which is yq-go (mikefarah), so the talos phase would fail there. Without the flag both print a document helm accepts, as the Cilium values line already relies on. - A cni-confdir pod left from an earlier run (Failed or Succeeded) made `kubectl apply` a no-op and the wait time out. The step deletes it first. - The Cilium HelmRelease comment named itself the only place Cilium is configured; the site patch's cilium-site ConfigMap is the other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5.5
|
10bb7c7893 |
feat(talos): site facts live in the site patches, and what reads them follows
What differs between sites moves out of cluster-wide config into the Talos site patches (site-homelab.yaml, site-qemu.yaml), and the consumers read it from there: - nodeIP.validSubnets lists IPv4 first, as the live nodes run. Talos keeps the order (siderolabs/net FilterIPs) and the first entry becomes the primary node IP; with fd02:46::/64 first, kube-apiserver exits with 'service IP family "10.96.0.0/12" must match public address family'. - etcd.advertisedSubnets (VLAN 46, a fact of the home network) moves unchanged from cluster.yaml to site-homelab.yaml; the rendered production config is byte-identical. - Pod routing (routingMode, autoDirectNodeRoutes, directRoutingDevice) and the native-routing CIDRs are a cilium-site ConfigMap per site patch, read by the Cilium HelmRelease via valuesFrom. bootstrap.sh merges it the same way: site values first, HelmRelease values on top. Production renders unchanged (helm template, TLS secrets excluded). - Each site patch states its node resolver list (ResolverConfig: k8s-gateway first, then the site's resolver) and k8s-gateway's upstream as a k8s-gateway-site ConfigMap, read by the k8s-gateway HelmRelease via valuesFrom. bootstrap.sh installs k8s-gateway right after the nodes are Ready, from the HelmRelease under the site values, and Flux adopts it: until k8s-gateway answers, every lookup waits out the node's 2 s read timeout and pods resolve nothing, Flux included. - bootstrap.sh creates Cilium's cni.confPath subdirectory with a hostNetwork pod per node (/etc/cni is read-only to the kubelet on Talos) and applies Multus right after Cilium; without it the kubelet reports "cni plugin not initialized" and nothing schedules on a cold cluster. - site-homelab.yaml labels its nodes net.renner.uno/pod-table=47 (table 47 exists only where the site routes pod traffic into it); k8s-1..3 label hardware.renner.uno/intel-gpu and net.renner.uno/iot-vlan, the hardware each machine has. The workloads select on these labels in later changes. Rollout order for production: the cilium-site and k8s-gateway-site ConfigMaps must exist before the HelmReleases reconcile (valuesFrom is not optional). Talos creates inline manifests on a config change but never updates them; changes go through talosctl upgrade-k8s. The node labels go on with talosctl patch, no reboot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5.5
|
222d0263a4 |
feat(navidrome): self-hosted music streaming on the bulk pool
Navidrome (Subsonic API) as the Spotify replacement for phone apps. - Library PVC on ceph-filesystem-bulk (RWX, 300Gi, mounted read-only); SQLite data on ceph-block (RWO). - Web UI behind a dedicated navidrome-gate Authentik proxy outpost; Navidrome trusts X-Authentik-Username (ExtAuth) and auto-creates the user on first SSO login (first user = admin). - /rest and /share routed straight to Navidrome with the identity header stripped, so Subsonic clients use their Navidrome password. - Namespace baseline CNP, gate egress CNP, Flux image automation, Velero daily schedule (09:15), bootstrap wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5.5
|
eb23d3ac6e |
feat(gitea): move Gitea into the gitea namespace (stage 2 of #878)
- kubernetes/apps/gitea: every object in namespace gitea; applied by a new gitea Flux Kustomization (clusters/homelab/gitea.yaml) instead of apps. gitea-allow-metrics-scrape dropped: the gitea-ns baseline already admits monitoring. - Rotators (flux-git, review-bot) and the Prometheus gitea job use gitea.gitea.svc. - bootstrap.sh waits for the gitea KS. Data: data-gitea-0 is not copied. The PV (reclaimPolicy Retain) is rebound to a PVC gitea/data-gitea-0 pre-created with volumeName, which the new StatefulSet adopts. gitea-tls is pre-issued in the gitea namespace so HTTPS has no certificate gap. Refs #878 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
f0701370db |
docs: describe the traffic-class split as built (#576)
Replaces the proposal, which was wrong before it was written down. It presented A, B and C as the option space and concluded pod-to-pod was blocked; policy routing was available the whole time and is what moved pod egress and then pod-to-pod. The file recorded a blind spot and gave it the authority of a checked-in document. Three of four classes are separated now, so this describes the built design rather than proposing one, and the PROPOSAL- name goes with it: which link each class uses, the rule order and why 980/981 match source and destination, where each table comes from, how the bird mesh renders itself from the API without repeating an address, and why the node's own CIDR is originated rather than read back from the kernel. Management is the remaining class. Its levers are listed, along with the two approaches the owner rejected — with the caveat that the same reasoning which produced that list also called pod-to-pod impossible, so it is a record of decisions and not an analysis to trust. Failure behaviour is stated because it is what makes the design safe to roll one node at a time: an empty table 48 falls through to main and pod traffic returns to VLAN 40, unseparated rather than broken. bootstrap.sh's reference follows the rename; its point was always that the QEMU/Hetzner sites use two segments, not the homelab's split. Refs #576 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Fable 5.1
|
193bcb362e |
fix(openbao): QuorumLost runbook text must not point at the recovery key
The description told on-call to try the recovery key as a last resort, while RUNBOOK Step 4 in the same change (and upstream: recovery keys are "purely an authorization mechanism") says it cannot unseal an auto-sealed server. Say what actually helps: fix the Secret, restart the pod. Also drop a duplicate root-token computation in restore_openbao. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
claude-botandClaude Fable 5.1
|
35bad376b8 |
feat(bootstrap): OpenBao restore and bring-up under the static seal
Follow-up to #825 (static-seal auto-unseal). The rebuild path and the operator docs still assumed shamir with a manual unseal per replica. bootstrap.sh - Required secrets: `seal-key` (pass show homelab/openbao/seal-key) replaces `unseal-key`. The snapshots are encrypted under the seal key, so it is the DR-critical secret now; the former unseal key is the recovery key and is not needed for a restore (upstream: recovery keys "are purely an authorization mechanism" and cannot unseal a server whose auto-seal is broken). - flux phase creates the `openbao-seal-key` Secret (namespace + Secret, before `flux bootstrap`) from the collected key. --from-literal via a substitution, because `pass show` appends a newline and OpenBao rejects a key that is not exactly 32 bytes of base64 ("unknown encoding for AES-256 key" — verified). An existing Secret with the same value is left alone; a different value is refused, since the data is written under whichever key OpenBao started with. - restore_openbao: `init -recovery-shares=1 -recovery-threshold=1`, root token from the output, wait for the self-unseal (init returns with the node sealed; the server's own retry loop unseals it within ~5 s), then `snapshot restore` WITHOUT -force: OpenBao verifies the snapshot was written under this server's seal key and refuses a mismatch by name, which is the one guard against restoring under a wrong Secret. -force skipped that check and would leave a server that can never unseal. The throwaway `unseal` step and its argv/tty discussion are gone — under an auto-seal there is no unseal key. - unseal_openbao -> bring_up_openbao: waits for spec.replicas pods to report sealed=false, walking the OrderedReady chain as before, and on timeout prints each still-sealed pod's `bao status` instead of a bare "timed out". Rehearsed locally with two OpenBao servers sharing a static key (file:// form, bao 2.5.5): init on an auto-seal returns an empty unseal_keys_b64, one recovery key and a root token and the server is unsealed within seconds; a restore of A's snapshot into fresh B (both with and without force) re-seals and reloads the root key via the auto seal ~5 s later ("done reloading root key using auto seal"), after which B serves A's data with A's root token. Restarting B under a different key leaves it sealed with "failed to decrypt keys from storage: message authentication failed". Docs and alerts: RUNBOOK-cold-rebuild Step 4 and MIGRATION-sops-openbao describe the auto-seal flow and the recovery-key semantics; the OpenBao alert descriptions and the snapshot CronJob comment no longer tell the reader to unseal by hand. vmtest-job comment updated. Not covered: a full VMTEST_FULL=1 rehearsal through the flux phase. The wiki records none ever reaching past the Flux bootstrap checkpoint, so that gap predates this change. Refs #805 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
claude-botandClaude Fable 5.1
|
93c0e33dac |
fix(ceph): OSD raw volume label must not contain "ceph"
All three OSD prepare jobs on the Hetzner rehearsal finished with zero OSDs: cephosd: skipping device "sdb1": ["Used by ceph-disk"] The partition is exactly what the machine config asked for (PARTLABEL r-ceph-osd, plain Linux partition type). ceph-volume rejects it anyway — its ceph-disk-member test is `"ceph" in PARTLABEL or PARTTYPE in ceph_disk_guids` (ceph_volume/util/device.py) — so any label containing "ceph" reads as a legacy ceph-disk partition and is never provisioned. Talos derives the label from the RawVolumeConfig name (r-<name>), so the name changes: ceph-osd -> osd-data, label r-osd-data. This is why the rebuild branch called it osd-data; the #798 review renamed it back for symmetry with the node label without this failure being known. Production is unaffected today: the MS-01 OSDs match on the by-id half of devicePathFilter and no r-ceph-osd partition exists there. The partlabel half is the migration target and would have failed on the first repartitioned drive. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
a4b659c965 |
feat(tooling): Hetzner throwaway site and rebuild-harness hardening
None of this is reconciled by Flux — bootstrap.sh, the vmtest Job and the Talos hcloud patches are operator tooling, applied only when a rebuild test runs. Zero effect on the live cluster from merging. New: an hcloudtest phase that provisions three cpx42 servers with private cluster/storage networks, a this-host-only firewall and one raw OSD volume per node, then hands over to the ordinary talos phase through the same four variables vmtest uses. Servers carrying the test label are adopted and rebuilt from the Talos snapshot instead of recreated — capacity, not cleanliness, is the scarce resource. Network ranges and volume sizes are derived from what the cluster itself declares (the site patch and the machine patches), never stated a second time. hcloudclean deletes everything by label. The vmtest Job clones from the GitHub mirror instead of in-cluster Gitea: a rebuild rehearsal cannot depend on the thing whose loss it rehearses, and a run died on resolving gitea.renner.uno before the first VM booted. bootstrap.sh hardening from a week of rehearsals: the OpenBao unseal loops until the StatefulSet converges instead of unsealing whichever pods existed at one instant, require_secrets takes a subset, secrets can be streamed into a waiting run, retries around talosctl bootstrap and server placement, and node waits print progress and refuse an expected count of zero. import-cluster-access.sh keeps its default AWS profile but stops overriding one the caller already chose. Known boundary, documented in site-hcloud.yaml: from master the hcloud site validates provisioning through cluster-Ready; the SAN path needs the follow-up rollout PR that renames the storage NIC by role. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 5
|
eec1f36c44 |
fix(bootstrap): honour a caller-supplied VM_STATE in vmtest
The vmtest Job passes VM_STATE=/work/vm to aim the qcow2 images at the
150Gi emptyDir it sizes for them. bootstrap.sh overwrote it
unconditionally, so the images landed on the container's writable layer
instead and the sizeLimit guard never applied to anything.
Default to the same path as before when the caller sets nothing, so a
plain `bootstrap.sh vmtest` is unchanged.
Two guards come with it, because the value is now caller-controlled:
* a length cap, since the provisioner puts a qemu monitor socket under
this directory and sun_path stops at 108 bytes. Too long used to be a
silent failure to boot; it is now an error. Both the old default (21
bytes) and the Job's /work/vm (8) are well inside it.
* a shape check, because cleanup_vmtest runs `sudo rm -rf "$VM_STATE"`.
That was safe by construction while the path was hardcoded and is not
once a caller picks it.
This does not by itself reduce drive wear — an emptyDir also lives on
/var, so both paths are on the same device. It restores the size guard
and makes it possible to point VM state at a different device later.
VMTEST_FULL and VMTEST_KEEP, which the Job also sets, are still not read
anywhere in this script. Left alone rather than guessed at: the Job is
applied out of band and is not in this repo, so the intended semantics
are not visible from here.
Refs #754
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
claude-bot
|
3b178068a8 |
fix(bootstrap): ask each node directly instead of waiting on cluster discovery
The formation wait failed with every node listed as unregistered, and the error
it now prints was the same one that broke the bootstrap call:
PermissionDenied: no request forwarding
`talosctl get members` went through the configured endpoint list rather than to a
specific machine, and apid will not forward it. It happened to work on some runs
and not others, depending on which endpoint was picked.
The second problem is worse and is not about VMs. Membership comes from the
discovery service, because the Kubernetes registry is disabled in the generated
config. So this step made a cold rebuild wait on discovery.talos.dev being
reachable, before the cluster being rebuilt exists — an internet dependency in
the middle of the disaster path, gating the bootstrap.
Neither is needed. What has to be true before etcd is bootstrapped is local: the
config took, the machine rebooted into it, and it is the machine we think it is.
The wait now asks each address directly for its hostname and expects the name its
own patch assigns — which also catches a patch landing on the wrong machine,
something the member list never checked.
The failure names the address, what it expected, and what it last answered.
|
||
|
claude-bot
|
ddd3dee23c |
test(bootstrap): make vmtest check the storage premise it depends on
Four nodes Ready says nothing about whether Ceph could use them. The CephCluster picks OSD nodes by a label out of the machine config and their device by the partition label RawVolumeConfig creates, so the phase now asserts both on every worker: the r-ceph-osd partition exists, and the Kubernetes node carries storage.renner.uno/ceph-osd. Checking this by hand afterwards does not work — the run's EXIT trap destroys the VMs as soon as it finishes, and that race was lost on the last attempt, leaving the partition confirmed on exactly one worker from an earlier failed run. Also drops the second statement of how many VMs there are. The node patch lists are the count now, rather than the flags and the progress line each carrying their own copy, and the addresses follow from the list instead of being spelled out. That list had already gone stale once when the topology moved from three control planes to one plus three workers. |
||
|
claude-bot
|
97b6461856 |
fix(bootstrap): send the etcd bootstrap to one control plane directly
The four-VM run reported:
error executing bootstrap: rpc error: code = PermissionDenied
desc = no request forwarding
and the cluster then sat idle. Nothing downstream noticed: `talosctl kubeconfig`
still succeeded, because Talos issues credentials as soon as it can rather than
when kube-apiserver serves, so the run carried on to the API wait as if it had
bootstrapped.
etcd bootstrap is the one call apid will not forward — reached through any node
other than the target it fails outright. `talosctl bootstrap` with neither -n nor
-e picks an endpoint from the configured list, and that list holds every control
plane, plus the workers in the vmtest phase. Whether it landed on the right node
was chance, in a real rebuild as much as on VMs.
Only visible because the previous commit stopped discarding this error.
|
||
|
claude-bot
|
94701ea236 |
fix(bootstrap): stop the talos phase from hiding why it failed
The four-VM run died on "Talos cluster didn't form" after 150s. The message was wrong twice over: the wait was too short, and it could not have told the difference between an incomplete member list and no node answering at all, because talosctl's error went to /dev/null. Membership here comes only from the discovery service — the Kubernetes registry is disabled — so a node registers after it has installed to disk and rebooted. Four machines do not finish that in 150s. The timeout is now 10 minutes, and the failure names the nodes that never registered, the last talosctl error, and the member list it actually saw. Three more of the same shape in this phase: `talosctl bootstrap` discarded every error. The `|| true` exists because re-running against an already-bootstrapped cluster is fine, but it hid genuine failures too, and the run then continued to break somewhere further on with nothing pointing back here. Anything that is not "already exists" is now reported. `talosctl kubeconfig` held its error while retrying and then failed with "Couldn't get kubeconfig" and no reason. The reason is now printed when the retries run out. The node-ready wait counted with `grep -c "Ready"`, which also matches NotReady — the substring is in it. So it waited for nodes to register, not to become ready, and would report success with every node still NotReady. It now matches the STATUS column whole. This is why the previous run's "3 nodes Ready" claim was not evidence of readiness; whether those nodes were in fact ready is unknown. |
||
|
claude-bot
|
b63440ee39 |
feat(ceph): let the machines say which disks are OSDs
The CephCluster named three hostnames and three drive serials. That tied the
storage layer to these three boxes as firmly as the Talos config used to be tied
to them, and for the same reason: it said in a Kubernetes manifest something only
the machine knows.
Now the machine says it, twice:
nodeLabels storage.renner.uno/ceph-osd which machines carry storage.
placement.osd.nodeAffinity reads it,
so useAllNodes can be true without
putting OSDs everywhere — that
nodeAffinity is the only thing
keeping them off diskless nodes.
RawVolumeConfig ceph-osd which disk, chosen by a CEL
expression per machine. The
partition surfaces at
/dev/disk/by-partlabel/r-ceph-osd
and that is what Rook consumes.
devicePathFilter carries both that partlabel and the OSD model for now. The
three production drives hold bluestore written straight onto the raw disk, with
no partition table and so no partition label to match; they are migrated
separately, one node at a time, and the by-id half goes when they are. Matching
the remaining half by model rather than by three individual serials already means
a drive swap is not a repo edit.
Resource requests now follow what the daemons use, measured on the running
cluster: mon ~420Mi against a 2Gi request, mgr ~650Mi against 4Gi, OSDs ~2.2Gi
against 4Gi — roughly 17GiB reserved cluster-wide and never touched. Limits stay
where they were, so burst headroom is unchanged; only the reservation moves.
Guaranteed QoS is given up deliberately, and priorityClassNames already pins
these to system-node-critical, which is what actually orders eviction. The OSD
request is set above measured use on purpose: an OSD grows into Ceph's
osd_memory_target, so 2.2Gi is a floor rather than a ceiling.
The vmtest phase becomes one control plane and three workers. Workers because
the QEMU provisioner gives the second disk only to worker machines and an OSD
needs one; three because the mons run with allowMultiplePerNode false and the
pools are replica 3 over failureDomain host. The apply loop now walks ALL_NODES
once instead of once per role, so NODE_ADDRS stays a single ordered list.
Not yet verified: whether Ceph actually reaches Ready on the VMs. That is the
point of the run this enables, and the memory budget is tight enough that it may
not — 2.5GiB plus three times 4GiB against 15.4GiB available.
|
||
|
claude-bot
|
9c5ed5dfcf |
feat(bootstrap): make the flux branch configurable so the phase can be rehearsed
Both bootstrap calls hardcoded --branch=master, which is why the flux phase could never be tried before it was needed: `flux bootstrap` commits its own sync manifests into the branch it reconciles, so running it wrote to the source a rebuild depends on. FLUX_BRANCH defaults to master and can be pointed at a throwaway branch. That branch has to exist first — `flux bootstrap` clones it and fails with `couldn't find remote ref` rather than creating it, which is not obvious and is why the comment spells the two commands out. Rehearsed against a scratch cluster to establish that: `flux bootstrap` clones the named branch, commits its manifests there, and installs the controllers. The waits after it need real disks and were not exercised. Nothing else about the phase changes, which is the point — a rehearsal that takes a different code path proves less than one that takes the same one. |
||
|
claude-bot
|
6e8d4cdfba |
feat(talos): make the config self-sufficient, and test it on three VMs
bootstrap.sh generated a config and then edited it with awk before applying it,
cutting out the HostnameConfig document `talosctl gen config` emits. That was
added in April when Talos 1.12 started emitting it, and it has been load-bearing
since: without it apid rejects apply-config outright with
static hostname is already set in v1alpha1 config
The document was never the problem. Naming the host in two schemas at once is:
machine.network.hostname and machine.features.stableHostname are the v1alpha1
spellings of what the generated document already says. Setting the name in the
document instead — with `auto: off`, which is otherwise mutually exclusive with
a static name — removes the conflict, and with it the awk.
That matters beyond tidiness. Post-processing the generator's output meant the
config could only be produced by this one script: anything else that ran
`talosctl gen config` against these patches, including `talosctl cluster create`,
got a config the nodes refuse. Which is why none of this had ever been tried
outside a real rebuild.
Two more values moved out of cluster.yaml for the same reason — they described
this network rather than this cluster, so a config generated for anywhere else
was wrong before it was applied:
kubelet.nodeIP.validSubnets now talos/patches/site-homelab.yaml, a third
layer between the cluster and the machine. It
pinned every node to 10.2.40.0/24 whether or not
it has an interface there.
cluster.controlPlane.endpoint removed outright. `gen config` takes the
endpoint as its second argument and bootstrap.sh
already passes it; restating it here overrode
whatever a caller had generated for.
With that, `./bootstrap.sh vmtest` provisions three throwaway QEMU VMs and runs
the talos phase against them unchanged. It is deliberately not a copy of that
phase: it sets NODES, NODE_ADDRS, SITE and K8S_API_HOST and falls through, so a
passing test says something about a rebuild rather than about itself. Talos
version and Image Factory schematic are read out of install.image in
cluster.yaml rather than named again.
It found two real defects on its first run, both fixed here:
The phase used the kubeconfig the moment `talosctl kubeconfig` returned, which is
when Talos can issue credentials — not when kube-apiserver is serving on the
endpoint the file points at. Helm hit `EOF` against a port nothing was listening
on. It now waits for `kubectl get --raw /version`, which needs the server;
`kubectl version` would have answered from the client and passed against nothing.
`echo` without -e printed raw escape codes instead of colouring the per-node
result.
TALOSCONFIG/KUBECONFIG are honoured for the access files instead of hardcoded
$HOME paths, unset resolving to the same two paths as before. That is what keeps
a test run from replacing the operator's production access with a VM's; access-
staging is skipped under vmtest for the same reason.
Verified: three VMs Ready on Cilium 1.20.1 from the HelmRelease, and all four
node patches generate and validate with no post-processing at all.
Not covered, and still only provable on hardware: Ceph (the provisioner gives
extra disks to workers only), the mgmt and v30iot VLANs, the Layer2 VIP, Flux,
and applying the three changed production patches to the real nodes.
|
||
|
claude-bot
|
c92aff9e75 |
docs(talos): rename registry-mirror.yaml to what it actually does
The name says the bootstrap depends on a registry mirror. Reading the patch list in the runbook, that is the obvious conclusion — and it would be a serious problem, because a cold rebuild has no mirror: an in-cluster one cannot exist before the cluster does. The file configures no mirror. It writes one containerd setting, discard_unpacked_layers = false, so that pulled layers stay in the content store and Spegel can advertise them to peers later. Spegel writes the per-registry hosts.toml itself at runtime; the file's own comment says a machine.registries.mirrors block is not needed, and there is none anywhere in talos/patches. Image pulls at bootstrap go straight to the real registries. So the alarm the name raises is false, but it is the name doing it, and the cost of that is someone deciding a rebuild is blocked when it is not. containerd-layer-retention.yaml describes the single thing inside. Renamed with git mv and the three references updated. Verified `talosctl gen config` still produces a config with the renamed patch in the list. |
||
|
claude-bot
|
cfba60afd0 |
feat(bootstrap): revive the talos phase and let it install the CNI
The Cilium install had become a block of shell in a runbook for a human to paste. That is the same rot as a pinned version — nobody runs it between rebuilds, so nothing tells you when it stopped working. It belongs in the script that already drives the rest of the bootstrap. The talos phase was unusable since #581: it passed cilium-cni.yaml whole, and that file's `$patch: delete` document fails against a config with no inlineManifests. The file is gone, so the failure is gone with it — reviving the phase was mostly deleting two lines that referenced it. It now installs Cilium between `talosctl bootstrap` and the node-Ready wait, which is the only place it can go: with `cni: none` the kubelet stays NotReady until a CNI exists, so the wait below would otherwise time out against a cluster that was never going to converge. Chart repo, version constraint and values are read out of helmrelease.yaml at run time, so the script cannot install something different from what Flux reconciles minutes later. helm takes the semver range verbatim. There is a guard on the extracted values — the HelmRelease is a two-document file and selecting the wrong one yields an empty values set that helm silently renders as chart defaults, which is a failure mode worth catching loudly rather than discovering in a datapath. `--wait` because the next step needs a working network, and `upgrade --install` so a re-run after a partial bootstrap converges instead of erroring. Verified as far as it can be without wiping a cluster: `talosctl gen config` with exactly the patch list the script now passes produces a config carrying cni: none and proxy: disabled, no Cilium manifests, and it validates for metal mode. The apply-and-bootstrap half is untested since the rewrite, and the runbook says so — it keeps the manual procedure as the thing to read first and to fall back on. |
||
|
claude-botandClaude Opus 5
|
9cc20251fd |
fix(edge): retire the legacy compose project before any role starts a container
Review on #474 found two blocking defects, both in exactly the place the --check verification is blind: command and shell tasks are skipped, not simulated, under check mode. So "verified with --check --diff" covered everything except the two things that would actually have broken. 1. Ordering: a real run would have aborted. The legacy-project teardown lived in power-meter-sml, which the play runs last. zwave-gateway runs before it and does `docker compose up -d` for a container named zwave-js -- while the old shared project still owns that name. Docker enforces container names globally, not per project, so the command fails with "Conflict: container name already in use". That is not the missing-USB-device string the task tolerates, so the task fails and the play aborts before ever reaching the cleanup. Moved to pre_tasks, which is its correct home anyway: it is a migration step for both services, and putting it inside either one meant whichever ran first hit the conflict. The play now also declares edge_base_dir and edge_legacy_compose in its own vars rather than borrowing them from a role's defaults, since pre_tasks must not depend on role default resolution. And it removes the now-orphaned zwavejs.env left at the old shared project root. 2. Dangling handler. The libeatmydata task still notified "Restart edge compose services", a handler that was deleted with the old role. Masked today only because `creates:` makes the task report ok -- but the role's own comment says to delete that .so when the Alpine base bumps, and the next time it does, Ansible would fatally error with "handler not found". Now notifies "Restart zwave-js". Swept every notify in the new roles against every defined handler; that was the only unresolved one. Also from review: kubernetes/bootstrap.sh referenced the deleted role. Repointed to ansible/docs/edge-host.md. Re-verified: ok=66 changed=25. The teardown moved from the power-meter-sml group into pre_tasks and gained the orphaned-env removal, so the count goes 5 + 5 + 14 -> 5 pre-existing + 6 migration + 14 header repointing. Nothing else moved. The limit is worth stating plainly: this re-run still cannot prove the teardown works, for the same reason it missed the bug. The first real apply is the test. Co-Authored-By: Claude Bot <claude-bot@renner.uno> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 4.8
|
91f5baf661 |
refactor(authentik): fold seed-authentik into the authentik KS — retire the seed layer (stage 2)
The last app in the seed->apps migration. Folds authentik's ak-pg CNPG backend
into the existing dedicated authentik KS and, since authentik was the final
holdout, fully retires kubernetes/seed/.
- git mv seed/authentik/{cluster,objectstore,scheduled-backup,barman-creds-
externalsecret,hetzner-creds-externalsecret}.yaml -> apps/authentik/, add to
apps/authentik/kustomization.yaml (authentik has no data PVCs — all state is
in ak-pg).
- authentik KS: drop 'dependsOn: seed-authentik', add
'dependsOn: infrastructure-cnpg-barman', add the CNPG Cluster healthCheckExprs.
- Delete clusters/homelab/seed-authentik.yaml and the now-empty seed/authentik/.
Seed-layer retirement cleanup (kubernetes/seed/ is now empty):
- git mv seed/RECOVERY.md -> kubernetes/RECOVERY.md (sibling to the MIGRATION-*
docs); fix its stale 'flux reconcile kustomization seed-authentik' example to
'authentik'.
- e2e-test.sh: drop the last seed-* readiness check (seed-authentik).
- bootstrap.sh: rewrite the obsolete 'per-app seed-<app> KSes' gating comment to
reflect that app KSes now own their CNPG DBs directly, and de-'seed' the
wait-description labels (the one-shot rsync/pg_restore seed Jobs are long gone).
- opentofu/object_storage.tf: fix the CNPG ObjectStore path comment
(kubernetes/seed/* -> kubernetes/apps/*).
Safe ownership transfer (nothing pruned): all 5 seed/authentik objects carry
prune: disabled (from stage 1 #407), so deleting seed-authentik skips them in GC
and the authentik KS adopts them in place. No CNPG restart, no app restart.
Restore-test passed first: ak-pg backup restored cleanly (25 users, 368 tables,
1225 MB) using the corrected RECOVERY.md recipe, with a clean no-leak teardown.
Verified: kubectl kustomize apps/authentik builds with Cluster/ak-pg + exactly
5 prune-protected; bash -n bootstrap.sh + e2e-test.sh OK; kubernetes/seed/ fully
removed; no stale seed file-path references remain.
Left intentionally: barman/hetzner ExternalSecrets still read OpenBao keys under
seed/authentik/... (vault paths, not repo paths).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
claude-botandClaude Opus 4.8
|
6900271f86 |
refactor(woodpecker): dedicated woodpecker KS, retire seed-woodpecker (stage 2)
ci/woodpecker/pr/pr-review Pipeline was successful
Restructures woodpecker into its own Flux Kustomization instead of splitting its
app across the catch-all apps KS and its DB across seed-woodpecker.
- Add clusters/homelab/woodpecker.yaml: dedicated KS over apps/woodpecker with
the infrastructure-cnpg-barman dep + CNPG Cluster healthCheckExprs (moved off
seed-woodpecker) so the CNPG concerns live here, not in the shared apps KS.
- Remove woodpecker/ from apps/kustomization.yaml (catch-all apps KS no longer
owns the woodpecker app objects).
- git mv seed/woodpecker/{cluster,objectstore,scheduled-backup,
hetzner-creds-externalsecret}.yaml + MIGRATION.md -> apps/woodpecker/, and add
the four DB manifests to apps/woodpecker/kustomization.yaml.
- Delete clusters/homelab/seed-woodpecker.yaml and the now-empty
kubernetes/seed/woodpecker/.
- bootstrap.sh: add a wait_kustomization woodpecker entry (it used to be covered
by 'wait apps'); e2e-test.sh: add woodpecker to the KS readiness checks.
Safe dual ownership transfer (nothing pruned): every object carries prune:
disabled from stage 1 (#394) — the app objects via commonAnnotations, the DB
objects per-file — so when the apps KS drops woodpecker/ and seed-woodpecker is
deleted, both GC sweeps skip these live objects and the new woodpecker KS adopts
them in place. No CNPG restart (Cluster is not a Deployment); no app restart
(manifests unchanged, only the owning KS label flips).
Verified: kubectl kustomize apps/woodpecker builds 17 objects (13 app + 4 DB
incl. wp-pg Cluster), all prune-protected; bash -n bootstrap.sh + e2e-test.sh
OK; wp-pg is the sole CNPG cluster in apps ns and hetzner-s3-creds is used only
by wp-pg-store; seed/woodpecker fully removed.
Stage 3 will drop the transient commonAnnotations (one more accepted app roll).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
claude-botandClaude Opus 4.8
|
2c8a91162c |
refactor(influxdb): move seed-influxdb to apps/influxdb (step 2/2)
ci/woodpecker/pr/pr-review Pipeline was successful
Step 2 of the InfluxDB seed->apps handoff. #388 already landed prune: disabled on the live HelmRelease/HelmRepository/ExternalSecret, so retiring seed-influxdb here cannot trigger a helm uninstall — GC skips the annotated objects and the new influxdb KS adopts them in place. - git mv kubernetes/seed/influxdb/ -> kubernetes/apps/influxdb/ (annotations from #388 travel with the files). - Add kubernetes/clusters/homelab/influxdb.yaml (KS 'influxdb', path apps/influxdb, deps infrastructure/crds/ceph). Delete seed-influxdb.yaml. Review fixes from the closed atomic PR (#387): - bootstrap.sh: wait on 'influxdb' instead of the deleted 'seed-influxdb' (would otherwise spin the full 600s timeout on cold bootstrap). - e2e-test.sh: swap the seed-influxdb readiness check for 'influxdb'. - Fix stale comment paths kubernetes/seed/influxdb/... -> apps/influxdb/... in network-policies/home-assistant-ns.yaml and apps/monitoring/ helmrelease-prometheus.yaml. - Trim the helmrelease.yaml comment that described the removed seed-job.yaml, and the ha-token-job.yaml 'at cutover moves to apps/influxdb' note (this is that cutover). Left intentionally: externalsecret.yaml key 'seed/influxdb/bootstrap' is an OpenBao KV path, not a repo path — moving it needs a vault-side secret migration, out of scope here. Verified: kubectl kustomize apps/influxdb builds all objects with the right prune/force annotations; bash -n passes on bootstrap.sh + e2e-test.sh; no remaining seed-influxdb references in the repo (only the OpenBao KV path). Co-Authored-By: Claude Bot <claude-bot@renner.uno> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 4.8
|
2aa0989a8a |
fix(bootstrap): wire GitHub break-glass source to break Flux<->Gitea loop
Flux pulls manifests from the in-cluster Gitea, which is itself Flux-managed (Ceph data, in-cluster DNS+Traefik), so a cold rebuild has no entry point. Give it one: - bootstrap.sh: FLUX_SOURCE=github selects 'flux bootstrap github' against the mirror (github.com/konairius/Homelab, branch master) instead of 'flux bootstrap gitea'; defaults to gitea so steady-state is unchanged. - kubernetes/RUNBOOK-cold-rebuild.md: single tofu->talos->flux->openbao->apps runbook, with the break-glass github bootstrap and the manual OpenBao snapshot-restore+unseal step wired in. The GitHub mirror is already kept current by Gitea's own native push-mirror (server-side, sync_on_commit=true) — no setup and no new credential required; the runbook just references it. OpenBao restore stays manual (unseal key out of CI, in pass). Refs #187 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
 k.rennerandClaude Opus 4.8
|
12c2668865 |
fix(dns): repoint k8s-gateway fallthrough off the decommissioned PVE DNS VIP
The renner.uno fallthrough forwarder still pointed at the Proxmox DNS HA VIP 10.2.40.53, which is decommissioned/unreachable. Every non-Ingress renner.uno name (device/service hostnames like udm.renner.uno) therefore dead-ended in SERVFAIL/timeout. A routine home-assistant pod restart on 2026-07-14 forced HA to re-resolve its device hostnames and left ~1374 entities unavailable / ~30 config entries stuck (ZHA, Shelly, UniFi, Bluetooth, cloud integrations). Point the fallthrough at the UDM (10.2.40.1 / fd00::1), which serves the split-horizon renner.uno zone (verified: it answers udm.renner.uno). Also scrub the stale 10.2.40.53 references in the CoreDNS comment, bootstrap.sh DNS_SERVER, and the Gatus monitor (which was pointlessly probing the dead VIP). A live hotfix has already been applied to the k8s-gateway ConfigMap to restore resolution; this makes it durable so Flux does not revert it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PwcQMntG9y9XRkNMPDq2fe |
||
|
k.rennerandClaude Fable 5
|
b4828b7e6c |
retire k8s-edge: remove zwave-js/sml2mqtt from cluster, VM 153, node patch
ci/woodpecker/push/deploy-infrastructure Pipeline was successful
The USB cutover to edge-1 is complete and verified (zwave-js serving HA with 12 nodes via ws://edge-1.iot.renner.uno:3000, sml2mqtt publishing, open3e polling 66 DIDs). Removes the home-assistant-zwavejs flux kustomization + app dir, the sml2mqtt manifests, the zwave-js-ui image automation, opentofu vm_153, the talos k8s-edge patch, and the worker from bootstrap.sh. Mosquitto users/ACLs stay — edge-1 uses them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
6fe33cbb96 |
traefik: migrate to bgp-pool (10.2.99.86 / fd99::86)
Traefik's LoadBalancer moves from the default-pool (10.2.40.86 / fd40::86, L2-announced) to the BGP-routed off-link pool. Every hostname behind traefik — every *.home.renner.uno, vscode.home.renner.uno, the cluster status page, paperless, immich, etc. — now reaches the ingress via the UDM's BGP-learned routes instead of L2 ARP on the mgmt LAN. This is the lever that takes vscode "fully working via the routed network" — the TLS + DNS access path goes through the same BGP route as the direct bgp-pool LB IPs. Same migration covers every other traefik-fronted service in one step. Companion updates: explicit `external-dns.alpha.kubernetes.io/target` annotations on the pve and test-browser IngressRoutes (Traefik doesn't populate IngressRoute.status.loadBalancer, so the target is hardcoded and has to track the LB IP); the bootstrap.sh / e2e-test.sh smoke-test IPs; a stale code comment in cilium-l2.yaml. The label/annotation pattern matches the ha-vscode migration: chart's `service.labels` sets `lb-pool: bgp` (single Service handles both TCP and UDP for HTTP/3, so one label is enough), and the `io.cilium/lb-ipam-ips` annotation pins the IPs deterministically. external-dns publishes the new IPs to Cloudflare + UniFi DNS providers on its next 60s reconcile cycle after the Service IP flips. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
9eb701e30b |
post-migration: drop nginx cruft across the repo
- 7 Ingresses: remove leftover nginx.ingress.kubernetes.io/* annotations (proxy-body-size / proxy-read-timeout / proxy-send-timeout / proxy-buffering). Traefik streams without a body cap; the websecure entrypoint idleTimeout is 3600s globally. - 5 CNPs: drop inert `app.kubernetes.io/name: ingress-nginx` selectors, keep traefik (or swap nginx→traefik for the paperless-ns east-west egress rule). Comments updated. - test-browser: remove the stale 10.2.40.80 hostAliases block (DNS resolves naturally to the traefik VIP now). - bootstrap.sh + e2e-test.sh: update local LB IP 10.2.40.80 → .86. No functional change for live traffic — all removed bits were either no-op on Traefik or referenced the retired nginx VIP. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
bfc885762c |
flux(seed): split monolithic seed KS into per-app seed-<app> KSes
Each `<app>` KS now dependsOn its own `seed-<app>` instead of the shared `seed` KS, so a slow sub-tree (immich-library-seed at ~26m) no longer blocks unrelated apps. Audiobookshelf / paperless / music-assistant / Authentik unblock at their own seed-Job completion (T+~13–17 vs T+~38–39 in pipeline #884). Old `seed` KS is suspended + prune:false during the migration window so SSA-driven ownership transfer to the new KSes doesn't risk pruning Cluster CRs (which are not prune:disabled-annotated). A follow-up commit deletes seed.yaml + seed/kustomization.yaml. bootstrap.sh wait-apps drops the monolithic `wait_kustomization seed`; per-app waits absorb the seed runtime (immich 1800s, audiobookshelf 900s, paperless/authentik 600s, music-assistant 300s) and a `seed-influxdb` wait (600s) covers the InfluxDB HR + token Jobs since InfluxDB has no separate top-level KS. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
3ec24ef20d |
feat(flux): install image-reflector + image-automation controllers
The flux bootstrap was originally run without --components-extra,
so only the four core controllers (source, kustomize, helm,
notification) were installed. The image-* CRDs and Deployments
weren't in the cluster, which meant the image-automation
manifests added in 5c5d39e/b5508d0 immediately broke the
infrastructure Kustomization with "no matches for kind ImagePolicy
in version image.toolkit.fluxcd.io/v1beta2".
Verified by:
kubectl -n flux-system get deployments
-> 4 controllers, no image-*
kubectl get crd | grep image.toolkit
-> empty
kubectl api-resources --api-group=image.toolkit.fluxcd.io
-> empty
Two changes:
1. Regenerated gotk-components.yaml with
`flux install --components-extra=image-reflector-controller,
image-automation-controller --export`.
Now ships all six controllers + their CRDs.
2. bootstrap.sh: same --components-extra flag added to the
`flux bootstrap gitea` call so the next rebuild-k8s run
produces a cluster that includes the image controllers from
the start, not as a one-off bolt-on.
After flux reconciles flux-system, the CRDs land, the new
Deployments come up, and the infrastructure Kustomization can
apply image-automation/ cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
k.rennerandClaude Opus 4.7
|
3063b54ab4 |
feat(seed): dedicated migration phase between infra and apps
ci/woodpecker/manual/rebuild-k8s Pipeline failed
Replaces the per-app race-y CNPG bootstrap.recovery + velero-bootstrap-restore
pattern with a single seed Flux KS that populates databases and PVCs from
legacy hosts before any app KS reconciles.
Changes:
- New kubernetes/seed/<app>/ tree owning, per app:
- CNPG Cluster CRs with bootstrap.initdb (was: bootstrap.recovery) +
enableSuperuserAccess so seed Jobs can connect as postgres
- DB seed Jobs: ssh legacy 'pg_dump -Fc' | pg_restore -d <db> against
the running cluster (was: ephemeral PG + barman-cloud-backup → S3)
- PVC seed Jobs: rsync/ssh+tar from legacy → app PVCs as standalone
Jobs (was: in-pod init containers, blocking the app Deployment's
progressDeadlineSeconds for hours on a 250 GiB rsync)
- All ObjectStores, ScheduledBackups, barman-creds, PVCs, seed-ssh
Secrets relocated here
- New top-level kubernetes/clusters/homelab/seed.yaml Flux KS:
dependsOn infrastructure-cnpg-barman, timeout 4h, healthCheckExprs
cover CNPG Cluster phase + native Job succeeded.
- Stripped from kubernetes/apps/<app>/: Cluster CRs, seed Jobs, init
container rsync blocks, ssh volumes, seed ServiceAccount references.
~454 lines deleted, ~33 added.
- App KSs (immich, authentik, paperless, audiobookshelf,
music-assistant, influxdb): dependsOn velero-bootstrap-restore →
dependsOn seed.
- bootstrap.sh wait-apps: new wait_kustomization seed (4h) before
per-app waits; per-app timeouts dropped to 60s since they're
near-instant once seed is Ready.
- Deleted kubernetes/clusters/homelab/velero-bootstrap-restore.yaml;
manifests stay in kubernetes/infrastructure/velero-bootstrap-restore/
for post-cutover re-add (it remains the cold-rebuild path once
legacy hosts are decommissioned and seed phase is removed).
Eliminates the documented race failure modes:
- cnpg-bootstrap-recovery-no-initdb-fallback (recovery Job pounds empty
bucket until seed lands → multiple Error pods)
- velero-bootstrap-restore-bsl-sync-race (Job iterates schedules in an
empty bucket for ~15 min)
- immich-server progressDeadlineSeconds blowup (250 GiB rsync init
exceeds the 600s default and Flux marks the Deployment Failed)
Seed is migration-only: removed entirely once all services have cut
over from legacy. At that point the Cluster + ObjectStore +
ScheduledBackup + PVC manifests move back into kubernetes/apps/<app>/
and velero-bootstrap-restore.yaml is re-added at the cluster level.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
k.rennerandClaude Opus 4.7
|
ae76682fc8 |
fix(immich-seed,bootstrap): grant schema usage + bump MA timeout
ci/woodpecker/manual/rebuild-k8s Pipeline failed
immich seed Job: pre-creating extensions wasn't enough — vectors and vchord create their own schemas (`vectors`, `vchord`) owned by postgres. pg_restore under SET ROLE immich then hits "permission denied for schema vectors" when CREATE TABLE face_search references vectors.vector(512). Add GRANT USAGE, CREATE on those schemas to TARGET_OWNER right after CREATE EXTENSION. bootstrap.sh wait-apps: pipeline 825 timed out at music-assistant (10m). On cold-start the wait timer for music-assistant starts while velero-bootstrap-restore is still reconciling (~15min), eating most of the 10m budget on the dependency. Bump to 30m (180 iters × 10s). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
2e6f7dbb3f |
feat(seed): per-app cold-start Seed Job populates Garage for CNPG bootstrap.recovery
ci/woodpecker/manual/rebuild-k8s Pipeline failed
Adds kubernetes/apps/{authentik,paperless,immich}/seed-job.yaml — a
single k8s Job per app that:
1. Idempotency-gates on `aws s3 ls s3://k8s-cnpg/<cluster>/` being
empty (any object means a prior seed or live cluster archived).
2. legacy-dump init container: ssh + pg_dump (or docker exec
pg_dump for paperless/immich which run their legacy DB in
docker-compose) → emptyDir.
3. main container: ephemeral Postgres (initdb + WAL archive_command
wired to barman-cloud-wal-archive) + pg_restore the dump +
barman-cloud-backup writes a base backup to s3://k8s-cnpg/<cluster>/.
The Cluster's bootstrap.recovery now has a producer: the seed Job
populates Garage; CNPG's recovery Job retries on empty bucket, then
succeeds once the seed lands the base backup + end_wal. No more
chicken-and-egg where the in-pod init container needed the cluster
already up.
In-pod init containers stripped of DB seed:
- authentik: removed entirely (no PVC seed needed). HelmRelease loses
serviceAccountName, initContainers, ssh volume.
- paperless / immich: rewritten to PVC rsync only, sentinel-guarded
(`/data/.seed-done`) so subsequent restarts skip the rsync. No
kubectl, no Backup CR creation, no serviceAccountName.
seed-rbac.yaml deleted from authentik / paperless / immich (the only
RBAC consumer was the in-pod init container's kubectl calls). influxdb
keeps its seed-rbac (its init container still creates Velero Backup CRs).
bootstrap.sh wait-apps timeouts bumped for cold-start reality:
apps 5m → 15m (Ceph mon/mgr/OSD bring-up on fresh cluster)
authentik 5m → 20m (seed Job + CNPG recovery from the seed)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
k.rennerandClaude Opus 4.7
|
69dbee5140 |
fix(bootstrap): drop || true on wait_kustomization + wait_certificates
ci/woodpecker/manual/rebuild-k8s Pipeline failed
wait-apps was burning the full ~5h timeout chain (sum of per-KS timeouts up to 1440x10s for immich) before reaching e2e even when authentik/paperless/immich were doomed by an empty CNPG bucket. Removing || true lets set -euo pipefail halt at the first KS timeout, surfacing the cold-start failure in minutes instead of hours. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
f1472104da |
feat(k8s): control-plane VIP at k8s.renner.uno (10.2.40.60)
Replace the single-node API endpoint (k8s-1.srv.renner.uno) with a Talos shared VIP — etcd-elected leader hosts 10.2.40.60 on its virtio NIC, so kubectl/Flux survive any single CP-node loss. - Per-CP patches add interfaces[].vip on virtio_net via deviceSelector - cluster.yaml: controlPlane.endpoint flipped, VIP IP + k8s.renner.uno added to machine + apiServer certSANs - bootstrap.sh: K8S_API_HOST drives talosctl gen-config and the kubeconfig server: rewrite - New UniFi A record k8s.renner.uno -> 10.2.40.60 Existing apiserver cert doesn't list the new SANs; rebuild-k8s or a Talos cert rotation is needed before kubectl-via-VIP works. Also drop vestigial hastate = \"started\" from the three CP VMs — the proxmox_vm module never wired hastate to a haresource (no-op line), and PCI passthrough on the Ceph OSD pins these VMs to their host anyway, so HA migration was never a real option. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
da550cf5e3 |
fix: surface 3 distinct failures #731 surfaced
ci/woodpecker/manual/rebuild-k8s Pipeline failed
The instrumented wait_kustomization helper from #731 finally let us see *why* paperless / monitoring / home-assistant Kustomizations were stalling. Three independent bugs: 1. monitoring stalled forever with PrometheusRule/monitoring/garage: dry-run failed: no matches for kind "PrometheusRule" in version "monitoring.coreos.com..." Chicken-and-egg: the rule's CRD comes from kube-prometheus-stack's HelmRelease in the same Kustomization, so Flux's server-side dry-run on the rule fails before the chart can install. Move the rule into a new `monitoring-rules` Kustomization that dependsOn `monitoring` — chart installs (CRDs and all) first, then the rule applies cleanly. 2. home-assistant stalled with Service/home-assistant-iot: namespace not specified iot-service.yaml was missing `metadata.namespace: home-assistant` while every other manifest in the dir has it. Added. 3. paperless stalled with Deployment/paperless/paperless status: 'F[ailed early due to ...] The truncated message was cut off at 120 chars in the per-iteration progress line so we couldn't see the actual reason. Bump truncation to 240 chars and dump full `flux get kustomization` + recent warning events on TIMEOUT so the next run gives a definitive answer instead of needing a second cluster trip. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
1209d1fa12 |
fix(bootstrap): wait for all cert-manager Certificates Ready in wait-apps
ci/woodpecker/manual/rebuild-k8s Pipeline failed
A Flux Kustomization that owns an Ingress with a cert-manager.io/cluster-
issuer annotation is marked Ready as soon as the Ingress + Certificate
manifests are applied. The actual ACME DNS-01 challenge runs asynchronously
afterward (~30s-2min per Certificate via Cloudflare), so e2e's TLS check
can hit the ingress-nginx fake default cert ("issuer=Acme Co, CN=Kubernetes
Ingress Controller Fake Certificate") for any Certificate still Issuing.
Add a `wait_certificates` poll at the end of wait-apps that blocks until
every Certificate cluster-wide reports Ready=True. Same progress-printing
pattern as wait_kustomization (state-change + 60s heartbeat). Budget is
15m, well above realistic Cloudflare ACME issuance times even with several
Certs in flight.
Closes the gap on #730's `dash.k.renner.uno fake cert` and grafana
`/api/health returns nginx HTML` failures (the Service backend was likely
fine but the Ingress was being served the default cert because the
Certificate hadn't finished issuing yet).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
k.rennerandClaude Opus 4.7
|
9fadd552a0 |
fix(bootstrap): bump wait-apps timeouts for monitoring/paperless/home-assistant
Pipeline #730's wait-apps still produced an incomplete cluster for e2e: the new wait_kustomization helper warned-and-continued on monitoring (10m) and paperless (15m) before they finished reconciling under the load of immich's parallel 250GB extract. e2e then caught the half- built state via observability + grafana TLS + paperless 503 failures. Bumping timeouts costs nothing on the happy path — Flux reconciles all Kustomizations in parallel, so the wait just observes; longer budgets only matter when something's actually slow. - monitoring: 60 → 180 (10m → 30m). kube-prometheus-stack pulls many images and stands up Prom + AM + Grafana + Loki + node-exporter + state-metrics; first install on a contended cluster routinely takes 15-25 min. - paperless: 90 → 180 (15m → 30m). Velero CSI snapshot restore for the data + media PVCs runs concurrently with all the other restores. - home-assistant: 90 → 180 (15m → 30m). Same Velero contention plus HA Core's native restore on first start. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
7009b7d03d |
fix(bootstrap): extend wait-apps coverage + add per-Kustomization progress
ci/woodpecker/manual/rebuild-k8s Pipeline failed
The wait-apps phase only blocked on `apps`, `authentik`, `paperless`, and `audiobookshelf`. After moving the music-assistant / immich / home-assistant seed bodies into pod initContainers (and given monitoring is what serves the Loki/Prom/AM/Grafana endpoints the e2e suite hits), wait-apps was returning success while immich was still mid-extract and kube-prometheus-stack hadn't issued certs yet — so e2e ran against a half-built cluster and flaked on observability + grafana TLS + immich pod still in Init:0/2. Refactor: - factor the per-Kustomization wait into a `wait_kustomization` helper that prints the elapsed time, current Ready/Message status on every state change, and a 60-s heartbeat so a long silent reconcile is still visible - add waits for: monitoring, music-assistant, home-assistant, immich - give immich the longest budget (1440 × 10s = 240 min) — matches the prior immich-seed Kustomization's 4h ceiling for the ~250GB library extract Order is cosmetic (Flux reconciles in parallel), so the call sequence is fast → slow to surface progress earlier in CI logs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
7a5234532b |
feat(registry-mirror): add docker.io pull-through cache (LXC 250)
Mitigates docker.io anonymous rate limits (100/6h per IP) that cause ImagePullBackOff during Talos+FluxCD cluster rebuilds. Local-disk-backed distribution/distribution:3.0.0 in an external LXC so the cache survives full cluster wipes; Talos containerd is pointed at it via machine.registries.mirrors and falls back to docker.io if unreachable. Only docker.io is mirrored; ghcr.io / quay.io / registry.k8s.io aren't rate-limited. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
da76201f1c |
fix(k8s-edge): Apply taint via kubelet register-with-taints
Reverts the bootstrap.sh kubectl-taint hack and fixes it properly in the Talos machine config. Talos machine.nodeTaints is silently ineffective on workers — NodeRestriction admission blocks post-join taint writes by worker kubelets. Kubelets CAN register with taints via the --register-with-taints flag, so pass it through kubelet.extraArgs and the taint lands at node registration instead of post-hoc reconcile. Also drops the workload-class nodeLabel (same NodeRestriction issue: arbitrary-namespace labels aren't kubelet-settable); only the kubernetes.io zone label stays since that one is on the allow-list. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
816651db39 |
fix(bootstrap): Reinforce worker taints via kubectl after node join
Talos nodeTaints in the per-node config patch don't always take effect on first kubelet registration (pipeline 508 saw k8s-edge come up with zero taints despite talosctl apply-config succeeding). Result: every stateful workload the scheduler cared about landed on the edge node — Grafana, Prometheus, Loki, Authentik pods, etc. Apply taints + labels via kubectl after the node is Ready, which survives future reconciliation regardless of how Talos applies the config patch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
36b8096777 |
chore(audiobookshelf): Extend wait-apps budget to 80 min
ci/woodpecker/manual/rebuild-k8s Pipeline failed
First measurement on a 70 GB library took ~70 min end-to-end: 17 min LXC tar + upload, 52 min S3 download + extract into PVCs. 30 min was too tight — e2e would fire before the restore job wrote its sentinel. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
904bcac70a |
feat: Migrate Audiobookshelf from LXC to K8s
ci/woodpecker/manual/migrate-audiobookshelf Pipeline failed
Mirrors the Paperless pattern: new namespace, RWX PVCs, OBC-based S3 migration bucket, trigger-backup-job that kicks a Woodpecker pipeline, restore-data-job that streams tarballs directly from S3 into PVCs. The Deployment uses an initContainer + sentinel file so the app never boots on an empty /config while the SQLite tarball is still extracting, and the restore job is idempotent (skipped when the sentinel exists). Also extends bootstrap.sh wait-apps + e2e-test.sh to cover the new Kustomization, OBC, TLS cert, and /healthcheck on audiobooks.k.renner.uno. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
125135d3fe |
ci: Split cluster bootstrap into phased pipeline steps
Break bootstrap.sh into three phases (talos, flux, wait-apps) that each CI step can invoke independently. Export cluster access files immediately after the talos phase so users have kubeconfig / talosconfig even if later phases fail. Pipeline steps: 1. recreate-vms — tofu replace of all 3 Talos VMs 2. talos-bootstrap — generate + apply Talos, bootstrap K8s, get kubeconfig 3. export-access — SOPS-encrypt + commit talos/kube config 4. flux-bootstrap — install flux, wait for infrastructure + Ceph 5. wait-apps — wait for apps/authentik/paperless to reconcile Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
42bd935e3a |
ci: Wait for paperless during cluster bootstrap
ci/woodpecker/manual/rebuild-k8s Pipeline failed
Paperless data migration pulls ~3.5GB from S3 on first boot; give it a longer window (15 minutes) than authentik before declaring the bootstrap complete. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
cdac4439d2 |
chore: Remove set -x tracing from bootstrap
Pipeline succeeded end-to-end on the next run — tracing no longer needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |