Review blocker: the Proxmox image-exists check derived TALOS_FILENAME
from version only, so bumping the schematic left the cached filename
unchanged — a future VM (re)provision would find the stale old-schematic
image and skip the download, silently reintroducing net.ifnames=0. Embed
the schematic hash in the filename (pipeline + opentofu/talos_image.tf,
kept in sync) so a schematic bump forces a fresh download.
Also reconciles stale comments the rename left behind: the MA reconcile
docstring (hostNetwork-era eth1 explanation), cluster.yaml kubelet nodeIP
(eth1->ens19), and the matter NAD's k8s-edge claim (edge isn't migrated
and isn't a current node; the pod is pinned off it anyway).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
Review finding: domain filters gate deletion as well as creation, so
dropping the suffix would orphan the retired k. records instead of
removing them. The filter entry stays until sync has deleted them all
(no resource carries a k. host, so desired state for the suffix is
empty), then gets dropped — the comment says exactly when. Also fixes
the stale excludeDomains mention and the '13 networks' miscount in the
NAT header comments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
Review finding: on (40) Server, (42) Ingress and (49) k8s Ceph the
cluster nodes sit on the same L2 as the client, so a DNATed flow's
reply bypasses the UDM conntrack (asymmetric) — and clients there are
the nodes themselves, which already resolve the public names in-cluster.
Their idx slots stay reserved. Also records the restapi provider in the
lock file and trims trailing blank lines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
Per review: no family-specific behavior. Every rule is the same
ADDRESS_AND_PORT specific-address shape (endpoint x port x network,
6 rules per network), matching the pre-existing hand-made rules'
structure. Drops the tofu-owned firewall groups and the v4-only
FIREWALL_GROUPS path entirely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
No UniFi Terraform provider models v2 NAT translation fields
(ubiquiti-community 0.55 none, filipowm #104 open, resnickio shell-only),
so the rules go through Mastercard/restapi against the controller's
/proxy/network/v2/api/site/default/nat endpoint, same API key as the
unifi provider.
Rule shapes follow the behaviorally verified matrix on Network 10.5.62:
v4 uses FIREWALL_GROUPS (tofu-owned address+port groups); v6 must use
ADDRESS_AND_PORT specific-address rules (4 per network) because v6
group-based DNAT silently drops the address match and rewrites all
traffic on the listed ports. rule_index is static per network to keep
indices collision-free and stable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
Both destroy applies (runs 1180/1181) died instantly with 'context
deadline exceeded' on the pre-destroy status GET while the PVE API
answers in <1s: the bpg provider reads timeout_delete from STATE at
destroy time, and the stage-1 import stored 0 for the timeout-less
shims — a zero-second context. Read paths carry their own timeout,
which is why import and plan were clean.
v2 declares all timeouts explicitly (one in-place, no-API update
writes them into state) and mirrors the real container attributes
(unprivileged=true is ForceNew — mirroring prevents a replace).
No ignore_changes this time: the timeout update must actually apply.
Flow: merge -> plan (must be in-place only) -> apply -> follow-up PR
deletes the file -> destroy apply.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Stage 1 (#268) adopted the never-tofu-managed rustfs containers into
state (import run 1173: all three imported, plan 'No changes').
Removing the shims makes the next apply destroy the containers —
guest lifecycle fully through tofu, per owner rule.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Owner rule: ALL guest lifecycle goes through tofu; no manual pct/qm
destroy, by anyone. The rustfs trio (800/801/802) was never
tofu-managed, so the #265 sweep could not reach it. Stage 1 adopts the
stopped containers into state (pipeline TOFU_IMPORT hook, commands in
the file header); stage 2 deletes this file and the apply destroys
them.
lifecycle ignore_changes=all + started=false keep the shims inert:
tofu must never reconcile or start these, only own their destruction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Owner directive: PVE runs ONLY the three k8s VMs. The proxmox-dns trio
(lxc 118/218/318), rustfs trio (lxc 800/801/802), fortress (vm 102)
and paperless (vm 111) are decommissioned; guest definitions are being
destroyed on PVE out-of-band.
Removed:
- opentofu: lxc_118_proxmox_dns.tf, vm_102_fortress.tf,
vm_111_paperless.tf, backup_jobs.tf (the vzdump job covered ONLY
destroyed guests — next tofu apply removes the job from PVE; k8s
data protection is Velero/CNPG)
- ansible: roles/proxmox-dns, install-proxmox-dns{,-ha}.yml,
check-chaos-testing.yml (tested failover of decommissioned DNS-HA
and postgres LXCs), proxmox_dns_vault.yml, rustfs_vault.yml,
[proxmox_dns_ha] inventory group
- monitoring: dns-servers scrape job in both the k8s HelmRelease and
the legacy prometheus-server template — resolves TargetDown
(dns-servers)
Closes#259
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
gitea (LXC 114), the woodpecker server (231) and the woodpecker agents
(910/911) were migrated into the cluster (Phase 2: Service flipped
ExternalName -> ClusterIP, now in-cluster StatefulSets/Deployment; see
kubernetes/apps/MIGRATION-gitea-woodpecker.md) and the Proxmox guests
decommissioned -- but their OpenTofu declarations were left behind with
onboot/start = true. An unscoped `tofu apply` (deploy-infrastructure.yml)
would attempt to recreate the retired stack, and backup_jobs.tf still
advertised the dead VMIDs as PBS-protected.
- delete opentofu/woodpecker_agents.tf, lxc_231_woodpecker.tf,
lxc_114_gitea_srv_renner_uno.tf
- drop VMIDs 114/231 and the 910.. agent loop from backup_jobs.tf
- remove the now-orphaned woodpecker_agents local + woodpecker_agent_count
variable (only those files referenced them)
- delete the dead .woodpecker/deploy-woodpecker-agent.yml and
install-woodpecker-server.yml pipelines (their ansible plays/roles no
longer exist)
REQUIRES `tofu state rm` of the four module resources BEFORE this merges
/ any apply runs, otherwise the plan destroy-plans them. See PR body.
Refs #199
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
object_storage.tf provisioned all four Hetzner buckets (CNPG WAL/base
backups, Velero PVC backups, cluster bootstrap secrets, tfstate) with
object_locking = false and no versioning, so a single leaked/misused S3
credential could irrecoverably delete backups and state.
Add minio_s3_bucket_versioning for all four buckets: a plain
DeleteObject no longer destroys data, the prior version survives until
purged by version ID.
object_locking is NOT flipped to true here: per the aminueza/minio
provider source, object_locking is only consumed on bucket Create, has
no Update handling and no ForceNew, so setting it on these
already-live buckets would be silently ignored -- not a real fix.
Enabling it requires destroying and recreating each bucket (full data
migration), which needs an operator decision; left as a follow-up
(see gitea issue #191 comments).
Also add minio_s3_bucket_lifecycle noncurrent-version expiration
(30 days) on the two high-churn backup buckets (cnpg, velero) to match
the retention window already enforced at the app layer (CNPG Barman
ObjectStore retentionPolicy, Velero Schedule ttl), so old S3 versions
don't accumulate indefinitely. secrets/tfstate buckets are low-churn
and left uncapped.
Fixes#191
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
captive.renner.uno has the identical problem just fixed for udm: UniFi
auto-serves only its A record (captive -> 192.168.8.1, the UDM console)
and forwards AAAA upstream, where the *.renner.uno wildcard returns
proxy.renner.uno. Pin the AAAA to the UDM management ULA fd00::1 so
v6-first clients reach the UDM instead of the public reverse proxy.
Same value as udm because both names resolve to the same UDM console
(192.168.8.1 over v4; fd00::1 is its v6, verified serving the controller).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The UniFi Network integration in Home Assistant intermittently drops to
setup_error with:
Cannot connect to host udm.renner.uno:443 ... TLSV1_ALERT_INTERNAL_ERROR
Root cause is IPv6, not TLS or the UDM. UniFi auto-generates only the A
record for the UDM's own hostname (udm -> 192.168.8.1) and forwards AAAA
queries to its upstream, where the *.renner.uno wildcard answers with
proxy.renner.uno (the public NetBird reverse proxy). HA is IPv6-first, so
it resolves udm.renner.uno to the proxy over v6 and gets its handshake
rejected (unknown SNI -> tlsv1 alert internal error). The integration
recovers only when resolution happens to land on the working A path.
Pin an explicit AAAA to the UDM's own management ULA fd00::1:
- verified serving the controller (cert CN=udm.renner.uno),
- a ULA, so it doesn't rotate like the ISP 2003:d6:* GUAs,
- reachable from the cluster via the node v6 default route (UDM ll gw).
With a local AAAA present, UniFi answers it itself instead of forwarding
upstream, so v6-first clients reach the controller. Because the record
matches the cert CN, TLS verification also passes.
Applied via the unifi_dns_record resource (UniFi local DNS) on tofu apply.
Watch the apply: if the UniFi provider/UDM rejects AAAA record types, this
is the point it surfaces.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The USB cutover to edge-1 is complete and verified (zwave-js serving HA
with 12 nodes via ws://edge-1.iot.renner.uno:3000, sml2mqtt publishing,
open3e polling 66 DIDs). Removes the home-assistant-zwavejs flux
kustomization + app dir, the sml2mqtt manifests, the zwave-js-ui image
automation, opentofu vm_153, the talos k8s-edge patch, and the worker
from bootstrap.sh. Mosquitto users/ACLs stay — edge-1 uses them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the k8s-edge Talos VM (vm_153) with a dedicated Arch host
running the hardware-bound services: zwave-js-ui and sml2mqtt via
docker compose (configs ported 1:1 from the k8s manifests), plus the
new open3e CAN bridge (SocketCAN via systemd-networkd, venv + systemd
unit, read-only datapoint set).
Driver for leaving k8s here: stock Talos ships CONFIG_CAN unset
(siderolabs/talos#7686), so the Vitocal/VX3 CAN adapter cannot work in
any pod without a custom kernel per release.
Base-install procedure, secret bootstrap (vault.yml from cluster
secrets) and per-service cutover checklist in roles/edge-host/README.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All 13 services run in k8s and were verified healthy/Running before
removal (audiobookshelf, immich, loki, prometheus, ical-proxy,
music-assistant, authentik full stack, home-assistant, influxdb [in
cluster as influxdb-influxdb2], grafana [kube-prometheus-stack]). These
stopped Proxmox guests were the PVE->K8s migration rollback net; with the
replacements serving 1-7 weeks, retire them.
Removed tofu modules (destroys the stopped guests + their stale DNS):
- vm_103_homeassistant, vm_108_influxdb, vm_109_grafana
- lxc_121_ical_proxy, lxc_212_audiobookshelf, lxc_225_immich,
lxc_226_loki, lxc_227_prometheus, lxc_229_music_assistant
- lxc_authentik_ha (130/131/132/133) + its stale authentik-db.renner.uno
-> 10.2.40.135 (dead keepalived VIP) DNS record
- backup_jobs.tf: drop VMIDs 103/108/109/121/130/131/132/133/212/225/226/227/229
Kept: fortress VM (102, not a migrated service) and paperless VM (111).
IRREVERSIBLE: applying destroys these guests and wipes their disks.
Review the plan before apply.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The netbird-ingress DaemonSet, netbird reverse-proxy, and Traefik have
fully superseded the old ingress tiers, which have been stopped (not
deleted) since the 2026-05-09 PVE cutover. Remove the dead code:
- Caddy HA: LXCs 232/233/234 (opentofu/ingress.tf), caddy-ha role,
install-caddy-ha + install-crowdsec-caddy plays, caddy cred vars,
proxmox_caddy_ha group_vars, build-caddy/deploy-caddy pipelines,
ci-caddy-builder image.
- cloudflared: LXC 117, cloudflared-lxc role, install-cloudflared play,
cloudflare_tunnel/variables_cloudflare disabled tofu stubs.
- netbird-gateway: LXC 260 (replaced by the in-cluster netbird-ingress
DaemonSet), netbird-gateway role + play.
- backup_jobs.tf: drop VMIDs 117/232/233/234/260 from the nightly vzdump
include list (those LXCs are being destroyed).
- netbird-server: point the setup-key hint at netbird-ingress/netbird-proxy.
Kept deliberately:
- crowdsec role (reusable; basis for the pending Traefik bouncer).
- netbird-client role + play (loki/prometheus LXCs are tagged
netbird_client, so it remains live architecture).
Note: applying the opentofu change will plan a DESTROY of stopped LXCs
117/232/233/234/260 — review the plan before apply.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
State already migrated (migrate-tfstate pipeline; 45 resources verified on
Hetzner). Repoint providers.tf backend (bucket/region/endpoint/path-style)
and the deploy-infrastructure backend creds from the nas-1 Garage S3 to
Hetzner. nas-1 is now fully decommissionable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Target bucket for migrating the OpenTofu remote state off the nas-1 Garage
S3 (s3.nas-1.renner.uno) to Hetzner Object Storage. State copy + backend
repoint follow in subsequent steps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The agents are unprivileged Proxmox LXCs whose ip_config had only an ipv4
block — no IPv6 at all. So no matter how Docker/Woodpecker were configured
for v6 (daemon.json ip6tables, WOODPECKER_BACKEND_DOCKER_ENABLE_IPV6), the
step containers had no v6 uplink to NAT onto, and deploys to v6-only hosts
(proxy-1) failed 'Network unreachable'. IPv4 NAT works in these LXCs, so v6
NAT will too once the LXC actually has v6.
Add an optional ipv6 block to the proxmox_lxc module (auto/dhcp/static) and
set ipv6_address = "auto" (SLAAC off v40srv) for the agents.
NOTE: `initialization` is in the module's ignore_changes, so this does NOT
apply to running agents on a normal apply — they must be RECREATED to pick
up v6 (they're tagged `transient`). Recreate one at a time to keep CI up,
then re-run install-woodpecker-agent.yml on each.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The migrate-* PVC-tarball workflow is retired, so renner-uno-k8s-migration
is no longer needed. Remove it; the (empty) bucket is destroyed by the
targeted apply. Remaining backup buckets: cnpg, velero, secrets.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First apply created k8s-cnpg/k8s-secrets/k8s-migration but k8s-velero
failed: "The requested bucket name is not available." Hetzner Object
Storage uses a global bucket namespace (virtual-hosted style → buckets
are endpoint subdomains), so generic k8s-* names collide with other
customers. Prefix all buckets with the domain (renner-uno-k8s-*) to stay
unique and consistent. The three already-created plain buckets are empty
and get recreated under the new names by the targeted apply.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
deploy-infrastructure's `tofu plan` was failing for everyone (not just
this branch): CI runs `tofu init -upgrade`, and the loose `~> 0.41`
constraint resolved to unifi 0.50.0+, which changed the dns_record `ttl`
schema from an integer to a Go duration string. Every plan then died on
the existing integer TTLs:
Error: Invalid Time Duration String Value
unifi_dns_record.renner_uno (ttl = 300)
unifi_dns_record.authentik_db (ttl = 60)
Bisected the provider: 0.49.0 is the last version that accepts integer
TTLs; 0.50.0 introduced the break. Pin to ~> 0.49.0 so the existing
config keeps validating. (A future bump past 0.49 needs those TTLs
rewritten to duration strings first.)
Also regenerates the lock: adds aminueza/minio, drops the stale unused
hashicorp/aws entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the offsite S3 backup target that decouples cluster backups from
nas-1's local Garage, so nas-1 can be powered off nightly to save power
(and as a genuine DR improvement: backups + bootstrap secrets no longer
share a box with, and survive a rebuild of, the cluster).
Hetzner has no API to generate S3 credentials (Console-only, confirmed
empirically against the live hcloud API), so provisioning is two-part:
the keypair is created by hand in the Console, then the aminueza/minio
provider creates the buckets (k8s-cnpg, k8s-velero, k8s-secrets,
k8s-migration) over the S3 API.
- object_storage.tf: minio provider (fsn1) + for_each bucket resources
- variables_hetzner.tf: endpoint/region/keypair vars + enable flag
- providers.tf: aminueza/minio ~> 3.33
- deploy-infrastructure.yml: wire TF_VAR_hetzner_s3_{access,secret}_key
from Woodpecker secrets into plan + apply
Bucket names mirror the current Garage layout so the cluster-side change
is only an endpoint/region/addressing swap (handled in follow-up commits,
staged paperless-first).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ~10k-entity HA install OOM-looped at its 4Gi cgroup limit: working set
sits ~3.9Gi, ran cleanly overnight, then tipped over on morning load spikes
into a tight OOMKilled restart loop (recorder unclean-shutdown markers every
few minutes). The legacy HAOS VM 103 that ran this same workload had 16Gi.
Raise k8s-edge (VM 153) memory 6144 -> 12288 so the HA pod can have an 8Gi
limit with headroom for the other pinned pods (matter-server, otbr, zwave-js,
music-assistant). Freed RAM comes from the now-stopped VM 103.
Apply order: tofu apply + edge node reboot must land before the pod limit
change so the node actually has the RAM.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RustFS was retired 2026-05-09 (per the comment in
opentofu-bootstrap/rustfs.tf). The S3 endpoint at s3.renner.uno:9000
has been unreachable since then, but a lot of references in code
remained — health checks pinged a dead endpoint, alerts fired
permanently, and the ansible role + playbook + tofu locals would have
broken the next time anyone tried to use them. Cleaning up:
DELETED (pure RustFS, no other consumers):
- opentofu/rustfs.tf.disabled
- opentofu/providers.tf.backup
- ansible/inventory/group_vars/proxmox_rustfs.yml
- ansible/plays/install-rustfs.yml
- ansible/plays/archived/migrate-rustfs-storage.yml
- ansible/roles/rustfs/ (whole role)
EDITED (RustFS sections removed):
- opentofu/variables.tf: drop rustfs_count
- opentofu/locals.tf: drop rustfs_instances local
- opentofu/backup_jobs.tf: drop the rustfs_count VMID-range
- kubernetes/apps/gatus/configmap.yaml: drop the dead s3.renner.uno
health check
- ansible/roles/gatus/templates/config.yaml.j2: same removal in the
ansible-managed gatus template
- ansible/roles/prometheus-server/templates/alert-rules.yml.j2:
drop the RustFSUnhealthy alert
- ansible/roles/grafana-observability/templates/alert-rules.yaml.j2:
drop the rustfs-unhealthy alert
- ansible/plays/check-chaos-testing.yml: drop the RustFS test block
and the rustfs_vip var
NOT TOUCHED (caddy-ha retirement scope, separate cleanup):
- ansible/roles/caddy-ha/defaults/main.yml (still has
caddy_ha_s3_endpoint = "s3.renner.uno:9000")
- ansible/plays/install-crowdsec-caddy.yml (still loads
vars/rustfs_vault.yml)
- ansible/plays/vars/rustfs_vault.yml (loaded by the play above)
- scripts/restore-authentik-db.sh (manual one-off restore script
that still references s3.renner.uno)
- opentofu-bootstrap/{rustfs.tf,variables.tf} (separate tofu
project with local state; cleanup needs manual apply)
The Woodpecker secrets tf_rfs_access and tf_rfs_secret are no longer
consumed by anything in the repo and can be removed manually from the
Woodpecker UI when convenient.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After the previous commit removed the aws_s3_bucket.{tfstate,migration,
caddy_certificates} resources, nothing in the .tf code references the
AWS provider that pointed at the decommissioned RustFS endpoint
(http://s3.renner.uno:9000). Removing the now-dead config:
- providers.tf: drop required_providers.aws and the provider "aws"
block (with the http://s3.renner.uno:9000 endpoint).
- variables.tf: drop aws_provider_access_key and aws_provider_secret_key
(only consumed by the removed provider block).
- .woodpecker/deploy-infrastructure.yml: drop the corresponding
TF_VAR_aws_provider_* env entries from both plan and apply steps
(sourced from tf_rfs_access/tf_rfs_secret secrets — those Woodpecker
secrets can be removed manually if unused elsewhere).
Backend "s3" config in providers.tf stays — it talks to the NAS-hosted
Garage instance (s3.nas-1.renner.uno) via AWS_ACCESS_KEY_ID/SECRET env
vars from tf_garage_* secrets, which is separate from the dead RustFS
provider being removed here.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both records were marked "will be created" in the deploy-infrastructure
plan despite being in state with concrete UniFi IDs earlier — meaning
they got deleted from UniFi externally and tofu's refresh dropped them
from state. Per direction, they should stay deleted rather than be
recreated.
- dns_renner_uno.tf: remove "emby" entry from renner_uno_records map
- vm_103_homeassistant.tf: empty the cnames list on the HA VM module
After this change the next plan should show no action on these (gone
from .tf, gone from state).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three aws_s3_bucket resources (tfstate, migration, caddy_certificates)
and the unifi_dns_record.s3_vip pointed at s3.renner.uno (10.2.40.101),
the now-decommissioned RustFS VIP. Tofu plan was hanging in CI because
the AWS provider in providers.tf still points at that dead endpoint and
the parallel refresh of the three buckets times out at TCP, holding the
plan open indefinitely (confirmed reproducibly across pipelines 909 and
910 — same stall point after the three aws_s3_bucket refresh-start log
lines).
The buckets themselves are gone with RustFS so there's nothing to
destroy via the AWS API — state rm is the right primitive here. The
follow-up pipeline run will pass:
TOFU_STATE_RM='aws_s3_bucket.tfstate aws_s3_bucket.migration aws_s3_bucket.caddy_certificates'
which calls tofu state rm on each before tofu plan, dropping the orphan
entries without an API call to the dead endpoint. unifi_dns_record.s3_vip
stays in state and gets a normal destroy via the (responsive) UniFi
provider.
The AWS provider block in providers.tf is unused after this but left in
place to avoid scope-creeping into the broader RustFS-leftover cleanup;
that's a separate PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Spegel now handles all in-cluster image mirroring via P2P pull-through
caching, so the standalone distribution/distribution LXC is redundant.
This commit:
- removes the OpenTofu module — next deploy-infrastructure pipeline
will plan a destroy for module.registry_mirror
- removes the ansible role, play, and install-registry-mirror.yml
Woodpecker pipeline (all only invoked during LXC create/update)
- strips --registry-mirror / --insecure-registry kaniko flags from
.woodpecker/build-ci-images.yml — CI builds now pull docker.io
directly (single rebuild fits well under 100 anonymous pulls/6h)
- updates the rook-ceph operator chart-default comment so it
references Spegel instead of the LXC
The Talos patch file kubernetes/talos/patches/registry-mirror.yaml
keeps its name (now contains the discard_unpacked_layers .part for
Spegel); bootstrap.sh references it by that name.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously only k8s-edge had a v30iot NIC, so any IoT-pool LoadBalancer lease that
elected a different node would GARP on the wrong interfaces and the LB IP would be
unreachable from devices on the IoT VLAN. Attaching v30iot to all k8s VMs lets Cilium L2
elect any node and still win ARP on v30iot from that node.
- opentofu/vm_15{0,1,2}_*.tf: extra_network_bridges = ["v30iot"]
- talos/patches/k8s-{1,2,3}.yaml: switch mgmt selector from `driver: virtio_net` to
`interface: eth0` (driver selector would now match both virtio NICs); add `interface:
eth1` for v30iot with routeMetric=4096 to keep the default route on eth0 (same pattern
as k8s-edge). X710 (i40e) keeps its driver selector and stays correct even if its
kernel name shifts to eth2 after the new virtio NIC inserts.
- infrastructure/post-crds/cilium-l2.yaml: drop nodeSelector pinning the iot policy to
k8s-edge — every node is now a valid announcer.
Apply path: deploy-infrastructure pipeline → tofu plan + manual apply (live virtio
hot-plug, no VM destroy); Talos picks up new NIC + machineconfig on next reconcile.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plan against my Caddy-retirement work surfaced two unrelated USB-
passthrough drifts on the HA VMs:
vm_103 (legacy HAOS, being shut down): tofu config still had the
stromzaehler + ZWA-2 USB devices declared, but live state has them
removed (rebinded to vm_153 k8s-edge at the HA-on-K8s cutover).
Plan was about to re-add them. Dropped the usb block.
vm_153 (running K8s edge node): block order and usb3 in config
didn't match live -- live has ZWA-2 first / stromzaehler second
with usb3=true; config had the inverse. Plan would have flipped
usb3 to false on a running VM (real change) and reordered. Updated
the config to match live so plan shows zero diff for vm_153.
No functional change: vm_103 stays stopped without the USB block;
vm_153's running USB passthrough is unchanged. This unblocks the
Caddy-retirement apply by leaving plan with only the intended scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pairs with the K8s file/photos commit. Empties every ingress_cnames
list still pointing at the (now stopped) Caddy VIP, so external-dns-unifi
can take ownership of the hostnames where a K8s Ingress is the new
owner, and stale hostnames simply stop resolving.
caddy_srv_renner_uno (ingress.tf) file/dir/caddy.renner.uno -> []
file: served by apps/file
dir, caddy: stale, no live backend
lxc_121 ical-proxy ical-proxy.renner.uno [K8s apps/ical-proxy]
lxc_212 audiobookshelf audiobooks.renner.uno [K8s apps/audiobookshelf]
lxc_225 immich photos.renner.uno [K8s apps/immich (alias)]
lxc_226 loki loki.renner.uno [K8s monitoring/loki-ingress]
lxc_229 music-assistant music.renner.uno [K8s apps/music-assistant]
vm_111 paperless paperless/docs.renner.uno [K8s apps/paperless]
Hostnames with no K8s replacement (services being retired entirely)
will stop resolving after apply:
lxc_227 prometheus prometheus.renner.uno
vm_108 influxdb influx/influxdb.renner.uno
vm_109 grafana grafana.renner.uno
Also removed the unifi_dns_record.caddy_vip A record
(ingress.renner.uno -> 10.2.45.100). Nothing CNAMEs at it any more,
and the keepalived VIP it pointed at goes away with the Caddy LXCs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DNS: removed pve/home/auth/dash from caddy_srv_renner_uno.ingress_cnames
in opentofu/ingress.tf. pve is now fronted by the new K8s ingress-nginx
path (apps/pve Service + EndpointSlice with the three cluster IPs);
home and auth have been served by ingress-nginx for a while and the
live UniFi A records are owned by external-dns-unifi -- the orphan
CNAMEs in the file were drift that would have been re-created on next
apply, breaking the working setup. dash backend is dead (502 via
Caddy, confirmed via curl).
Compute: every Proxmox guest under tofu management is now flipped to
stopped/onboot=false, except LXC 250 (registry-mirror) which keeps
flux/woodpecker image pulls working:
LXCs (start=false, onboot=false, hastate="stopped"):
117 cloudflared -- replaced by K8s cloudflared Deployment
121 ical-proxy -- replaced by K8s apps/ical-proxy
212 audiobookshelf -- replaced by K8s audiobookshelf
225 immich -- replaced by K8s immich
226 loki -- retired
227 prometheus -- replaced by K8s monitoring
229 music-assistant -- replaced by K8s music-assistant
232/233/234 caddy -- this whole retirement work
260 netbird-gateway -- retired
130/131/132/133 authentik HA -- replaced by K8s authentik
VMs (vm_state="stopped", onboot=false):
102 fortress
103 home-assistant -- replaced by K8s home-assistant
108 influxdb
109 grafana -- replaced by K8s monitoring
111 paperless -- replaced by K8s paperless
Module change: removed `started` from the proxmox_vm module's
ignore_changes list. The previous behaviour ("Don't force start/stop
state") meant tofu would silently drop vm_state changes, leaving the
five VMs above running indefinitely. The trade-off is that any VM
manually stopped for maintenance (e.g. a K8s node) will be auto-
restarted on the next tofu apply -- be aware.
`hastate` is dead-code in both proxmox_lxc and proxmox_vm modules
(declared as a variable, not consumed anywhere). The hastate="stopped"
edits are documentation only -- Proxmox HA state still has to be
managed via the Proxmox UI / pve-ha-manager / API. If pve-ha is
holding any of these in started state, it will fight the tofu stop.
LAN-side CNAMEs on the per-LXC ingress_cnames lists (e.g.
photos.renner.uno on lxc_225 immich) are intentionally left alone
in this commit -- they'll either be auto-cleared by external-dns-unifi
once the K8s Ingress claims the name, or fall stale pointing at a
dead Caddy VIP. Cleaning those up is a follow-up sweep.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion to the K8s-side commit. Removing the CNAME lets
external-dns-unifi take ownership of ci.renner.uno on the next
reconcile and write an A record pointing at the ingress-nginx LB
(10.2.40.80) instead of the Caddy VIP (10.2.45.100).
Apply via the renner_homelab tofu pipeline; agents are unaffected
because they target woodpecker.srv.renner.uno:9000 directly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion to the K8s-side commit that put gitea behind ingress-nginx
via an ExternalName Service. Removing the two CNAMEs (gitea.renner.uno
and git.renner.uno -> ingress.renner.uno) lets external-dns-unifi take
ownership of those names on the next reconcile and write A records
pointing at the ingress-nginx LB (10.2.40.80) instead of the Caddy VIP
(10.2.45.100).
Apply path: tofu plan + apply on the renner_homelab pipeline. After
apply, external-dns-unifi reconciles within ~1 min; LAN clients pick
up the new A records as their resolvers' caches expire (UniFi DNS
TTL=300s by default for upserted records).
Caddy still has its own Caddyfile entries for these hostnames -- those
become dead config once DNS no longer points at the Caddy VIP. Cleaning
them up is a separate ansible-side commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
target_node flips from the `saturn` placeholder to `neptune`: VM 103
(legacy HAOS) currently holds the ZWA-2 + stromzaehler USBs on neptune,
so co-locating k8s-edge VM 153 on the same host means at cutover the
USBs rebind from VM 103 → VM 153 in-place with no physical move.
USB block declared on VM 153 ahead of the cutover apply. Until VM 103
is destroyed, the USB devices are still claimed there; this declaration
is therefore not safe to apply in isolation — it lands at the final-
migration `tofu apply` after VM 103 destroy releases the devices.
k8s-edge also stays a Proxmox VM (no bare-metal pivot) — decision
recorded inline + in wiki note k8s-edge-target-node-neptune-decision.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- opentofu: attach Proxmox `x710-port1` mapping as hostpci1 on the
three K8s controlplane VMs (jupiter/saturn/neptune).
- talos patches: add a second Device per node selecting the i40e
driver, with static /29 addresses (10.99.0.1/2/3), MTU 9000, no
routes — cluster-internal only.
- CephCluster: switch to `provider: host` and declare `addressRanges`
for both public (10.2.40.0/24) and cluster (10.99.0.0/29). Rook
v1.18 docs are explicit that the default CNI provider ignores
addressRanges, so host networking is required to consume the new
link for OSD↔OSD replication.
Activates on the next rebuild-k8s pipeline run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both NICs on k8s-edge default-DHCP at the Talos-default routeMetric=1024,
and the IoT VLAN's DHCP server also returns a default gateway, so two
same-priority default RouteSpecs landed (10.2.40.1 via eth0, 10.2.30.1
via eth1). Kernel installed one and rejected the other with EEXIST;
Talos's RouteSpecController retried the loser every ~30-70s, spamming
the controller-runtime log on every k8s-edge boot.
Talos v1.12.6's DHCPOptions exposes only routeMetric/ipv4/ipv6/duidv6
(verified against pkg/machinery/config/types/v1alpha1/v1alpha1_types.go
at the v1.12.6 tag) so the lever is metric. Bumping eth1's to 4096
keeps both defaults at different metrics — eth0 wins egress at 1024,
eth1 is a never-used fallback. Connected /24 on eth1 is unaffected, so
Cilium L2-IPAM (HA-iot/mosquitto-iot announcements), OTBR/matter-server
macvlan parents, and music-assistant's host-net L2 reach to renderers
keep working.
Also rewrites the opentofu comment for the second NIC to capture both
load-bearing tenants (music-assistant L2 locality for audio-sync
latency; OTBR/matter-server macvlan parent for Thread/Matter L2
presence) instead of citing the discovery protocols as the cause.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Container registry blob storage filled the gitea LXC's rootfs.
build-ci-images pipeline 841 had three of four kaniko image pushes
fail with HTTP 500 on the gitea v2 registry endpoint; gitea host
journal showed "no space left on device" across alloy + tmp writes
at the same instant.
4G is way too tight for a registry with the ci-{ansible,k8s,tofu,
caddy-builder} family at ~200-500MB each, especially with kaniko
keeping the previous date-tagged copy until GC. 16G gives headroom
for several months of date-tagged image churn before the next bump.
bpg's proxmox_virtual_environment_container does in-place rootfs
resize for LXC (`pct resize`), no restart needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 832 (build-ci-images) ran 4 kaniko builds in parallel
across 2 agents (woodpecker_agent_count=2). build-ansible got OOM-
killed — 2 GiB per agent split across 2 concurrent kaniko jobs is
~1 GiB per build, and ansible + collections + pip deps under
kaniko's tmpfs layer cache exceeded that.
4 GiB per agent gives ~2 GiB per concurrent build, which has
enough headroom for the largest of the four images. If we ever
hit the same wall again the next bump is doubling the agent
count, not the memory — kaniko parallelism doesn't help much
beyond ~2 GiB.
BPG's proxmox_virtual_environment_container does in-place memory
updates for LXC (cgroup-level, no restart), so the apply is safe
to run while the agents are servicing other jobs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- vm_113_netbox.tf removed. Apply 797 hit a bpg validator error on
the netbox in-place update ("Defined disk interface not supported.
Interface was , but only [ide sata scsi virtio] are supported"),
but the resource is being decommissioned anyway, so skip the
schema fix and let the next apply destroy the VM.
- backup_jobs.tf: prune VMIDs whose .tf files were deleted earlier
this session (112 ha-dev, 113 netbox, 124 kasm, 125 dev, 223 ollama,
235 wiki, 236 owncloud, 240 vault). Keeping them in the include
list would error on the next vzdump cycle once the VMs are gone.
Phase 4's cutover to Garage made backend + aws-provider share the
same AWS_ACCESS_KEY_ID env var, which now points at Garage. The aws
provider talks to RustFS (still — for caddy_certificates / migration /
tfstate buckets), and Garage credentials don't authenticate against
RustFS, so plan 794 regressed: refresh of the three aws_s3_bucket
resources returned "doesn't exist" → plan proposed to recreate them.
Give the aws provider explicit credentials via two new TF_VARs sourced
from the still-existing tf_rfs_* Woodpecker secrets. Backend keeps the
env-var path with Garage creds; aws provider gets RustFS creds via
TF_VAR. No state shape change.
This is a transitional separation. When RustFS is fully decommissioned,
the aws provider's endpoint also moves to Garage and the two cred
sources can collapse back together.
Phase 4 of the migration off RustFS. State file was copied + sha256
verified by pipeline 793. Switch:
- providers.tf backend "s3": endpoint http://s3.renner.uno:9000 →
https://s3.nas-1.renner.uno, region main → garage.
- deploy-infrastructure.yml (validate-secrets / plan / apply) and
rebuild-k8s.yml (recreate-vms): tf_rfs_access/tf_rfs_secret →
tf_garage_access/tf_garage_secret.
The aws provider block in providers.tf still points at RustFS — that
manages aws_s3_bucket.caddy_certificates which lives on RustFS, and
migrating that bucket is a separate concern.
This push will auto-trigger deploy-infrastructure plan against Garage;
expected to produce the same plan output as pipeline 787 (cleanup-only
baseline) plus the still-unfixed 8-resource provider misread, since
the actual rm+import payload couldn't persist last time and needs to
be re-run with a writable backend.
pvesh /cluster/resources confirms neither VMID is in the cluster
anymore. Same shape as the prior 5-file cleanup: refresh marks them
"has been deleted", removing the .tf lets state drop them cleanly on
next apply.
Leaves 8 misread resources still in scope (gitea, audiobookshelf,
authentik_app_1/2, authentik_db_1, loki, registry_mirror, wp-agent-2),
all now confirmed migrated to nodes other than their .tf
target_node — fix is rm+import in a single pipeline run with current
node mapping.
Earlier theory was that 0.104 broke reads for 10 resources. User
clarified: those resources had real out-of-band changes (storage moved,
HA migrations between nodes) — state is stale relative to where the
resources actually live now, so the provider's `<node>/<vmid>` read
404s. Pinning the provider does not fix that; the state itself needs
to be re-imported from current locations.
Putting the constraint back to `~> 0.95` so init-upgrade keeps pulling
the latest within 0.x. The 5-file resource cleanup stands.
Plan 785 failed: 0.95.x doesn't have `proxmox_backup_job` (added in
0.96.0), so backup_jobs.tf wouldn't compile. The checked-in
.terraform.lock.hcl shows 0.96.0 as the last-good version, which the
pipeline silently moved off of via `tofu init -upgrade`. Bumping the
floor to 0.96.0 keeps the resource available while still excluding
the 0.104 read regression.
claude-botandClaude Fable 5
k.renner