konairius/homelab
10
0
Fork 0
Code Issues 11 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

[recoverability][medium] OpenTofu state drift: retired Woodpecker/Gitea LXCs would be recreated by an unscoped apply #199

New Issue
Closed
opened 2026-07-14 09:47:58 +00:00 by claude-bot · 2 comments
claude-bot commented 2026-07-14 09:47:58 +00:00
Member
Copy Link
Copy Source

Finding

opentofu/woodpecker_agents.tf (LXC 910/911), lxc_231_woodpecker.tf (231), and lxc_114_gitea_srv_renner_uno.tf (114) still declare the retired standalone LXCs as live resources (onboot/start true). The services moved in-cluster; the LXCs were (per migration doc) destroyed on Proxmox WITHOUT tofu state rm. deploy-infrastructure.yml can run 'tofu apply' on the whole root module.

Verified

The .tf files exist on origin and declare the resources; backup_jobs.tf still lists VMIDs 114/231. Whether the Proxmox guests were actually destroyed is not verifiable from repo.

Impact

An unscoped tofu apply would attempt to RECREATE the retired Docker CI stack (or, if state still tracks them, keep managing ghosts). Backup_jobs.tf also gives false confidence that 114/231 are PBS-protected.

Fix

Reconcile state (import or state rm to match reality), delete the three .tf files and the stale VMIDs from backup_jobs.tf, and remove the dead .woodpecker/{deploy-woodpecker-agent,install-woodpecker-server}.yml pipelines (their plays no longer exist).

Automated security review 2026-07-14. Files verified present; live Proxmox state unverified.

## Finding opentofu/woodpecker_agents.tf (LXC 910/911), lxc_231_woodpecker.tf (231), and lxc_114_gitea_srv_renner_uno.tf (114) still declare the retired standalone LXCs as live resources (onboot/start true). The services moved in-cluster; the LXCs were (per migration doc) destroyed on Proxmox WITHOUT tofu state rm. deploy-infrastructure.yml can run 'tofu apply' on the whole root module. ## Verified The .tf files exist on origin and declare the resources; backup_jobs.tf still lists VMIDs 114/231. Whether the Proxmox guests were actually destroyed is not verifiable from repo. ## Impact An unscoped tofu apply would attempt to RECREATE the retired Docker CI stack (or, if state still tracks them, keep managing ghosts). Backup_jobs.tf also gives false confidence that 114/231 are PBS-protected. ## Fix Reconcile state (import or state rm to match reality), delete the three .tf files and the stale VMIDs from backup_jobs.tf, and remove the dead .woodpecker/{deploy-woodpecker-agent,install-woodpecker-server}.yml pipelines (their plays no longer exist). --- _Automated security review 2026-07-14. Files verified present; live Proxmox state unverified._
claude-bot commented 2026-07-14 10:10:04 +00:00
Author
Member
Copy Link
Copy Source

PR opened: gitea_admin/homelab#232 (fix/issue-199-tofu-state-drift, off current origin/master).

Verified against the repo before acting:

  • The three .tf files still declared the retired guests (114/231/910/911, onboot/start = true); backup_jobs.tf still listed 114/231 (and, via local.woodpecker_agents, 910/911) as PBS-protected.
  • The migration is real and complete on the k8s side — kubernetes/apps/{gitea,woodpecker}/service.yaml are flipped ExternalName -> ClusterIP (Phase 2), so gitea/woodpecker now serve from in-cluster StatefulSets/Deployment, not the LXCs. Per MIGRATION-gitea-woodpecker.md the guests were then pct destroy-ed.
  • The two dead pipelines reference ansible/plays/install-woodpecker-{agent,server}.yml + roles/woodpecker-*, all of which are already gone — they can only fail.

What the PR does: deletes the 3 .tf files + the 2 dead .woodpecker pipelines; drops 114/231 and the 910.. agent loop from backup_jobs.tf; removes the now-orphaned woodpecker_agents local + woodpecker_agent_count var (nothing else referenced them). tofu fmt clean; no dangling references remain.

The state must be reconciled BEFORE this merges / before the next apply (the PR body has the full runbook). Run against the live backend:

tofu state rm \
  'module.gitea_srv_renner_uno' \
  'module.woodpecker_srv_renner_uno' \
  'module.woodpecker_agents["910"]' \
  'module.woodpecker_agents["911"]'

(module resource type is proxmox_virtual_environment_container.this; removing the whole module address is intentional). If the .tf is deleted while these remain in state, an unscoped tofu apply will destroy-plan them. I did not touch live tofu state or run any apply.

Blockers — need your confirmation:

  1. Confirm the guests are actually gone: pct list on neptune (114/911) and jupiter (231/910). The repo can't prove the pct destroy ran.
  2. Confirm you want state rm (drop from management) rather than re-import.

Minor, out of scope: opentofu/scripts/validate_plan.py still hard-requires a gitea container, but it isn't wired into any pipeline (dead code), so it won't block CI.

PR opened: gitea_admin/homelab#232 (`fix/issue-199-tofu-state-drift`, off current `origin/master`). **Verified against the repo before acting:** - The three `.tf` files still declared the retired guests (114/231/910/911, `onboot`/`start = true`); `backup_jobs.tf` still listed 114/231 (and, via `local.woodpecker_agents`, 910/911) as PBS-protected. - The migration is real and complete on the k8s side — `kubernetes/apps/{gitea,woodpecker}/service.yaml` are flipped `ExternalName -> ClusterIP` (Phase 2), so gitea/woodpecker now serve from in-cluster StatefulSets/Deployment, not the LXCs. Per `MIGRATION-gitea-woodpecker.md` the guests were then `pct destroy`-ed. - The two dead pipelines reference `ansible/plays/install-woodpecker-{agent,server}.yml` + `roles/woodpecker-*`, all of which are already gone — they can only fail. **What the PR does:** deletes the 3 `.tf` files + the 2 dead `.woodpecker` pipelines; drops 114/231 and the 910.. agent loop from `backup_jobs.tf`; removes the now-orphaned `woodpecker_agents` local + `woodpecker_agent_count` var (nothing else referenced them). `tofu fmt` clean; no dangling references remain. **The state must be reconciled BEFORE this merges / before the next apply** (the PR body has the full runbook). Run against the live backend: ```sh tofu state rm \ 'module.gitea_srv_renner_uno' \ 'module.woodpecker_srv_renner_uno' \ 'module.woodpecker_agents["910"]' \ 'module.woodpecker_agents["911"]' ``` (module resource type is `proxmox_virtual_environment_container.this`; removing the whole module address is intentional). If the `.tf` is deleted while these remain in state, an unscoped `tofu apply` will destroy-plan them. I did **not** touch live tofu state or run any apply. **Blockers — need your confirmation:** 1. Confirm the guests are actually gone: `pct list` on neptune (114/911) and jupiter (231/910). The repo can't prove the `pct destroy` ran. 2. Confirm you want `state rm` (drop from management) rather than re-`import`. Minor, out of scope: `opentofu/scripts/validate_plan.py` still hard-requires a gitea container, but it isn't wired into any pipeline (dead code), so it won't block CI.
claude-bot commented 2026-07-14 17:33:48 +00:00
Author
Member
Copy Link
Copy Source

Live check before merging #232 — the retired guests are NOT destroyed, only stopped. SSH to the nodes:

  • jupiter: 231 woodpecker.srv (stopped), 910 wp-agent-1 (stopped)
  • neptune: 114 gitea.srv (stopped), 911 wp-agent-2 (stopped)

The migration's pct destroy step was never completed. This changes the premise: these aren't orphaned drift — the guests still exist and match tofu state. Consequences:

  • Merging #232 (removing them from config) + tofu apply would destroy the stopped LXCs.
  • tofu state rm instead would leave them as untracked stopped LXCs on Proxmox.

Decision needed before #232 merges: (a) fully decommission → pct destroy 114 231 910 911, then tofu state rm the 4 modules (or let the config deletion destroy them); or (b) keep them stopped as a fallback — e.g. LXC 114 (gitea) as a break-glass for the in-cluster Gitea per #187 — in which case hold #232 until you're ready. Also: the workstation pass/env has no Hetzner S3 backend creds, so the tofu state rm/apply needs to run where those creds live regardless.

**Live check before merging #232 — the retired guests are NOT destroyed, only stopped.** SSH to the nodes: - jupiter: `231` woodpecker.srv (stopped), `910` wp-agent-1 (stopped) - neptune: `114` gitea.srv (stopped), `911` wp-agent-2 (stopped) The migration's `pct destroy` step was never completed. This changes the premise: these aren't orphaned drift — the guests still exist and match tofu state. Consequences: - Merging #232 (removing them from config) + `tofu apply` would **destroy** the stopped LXCs. - `tofu state rm` instead would leave them as untracked stopped LXCs on Proxmox. **Decision needed before #232 merges:** (a) fully decommission → `pct destroy 114 231 910 911`, then `tofu state rm` the 4 modules (or let the config deletion destroy them); or (b) keep them stopped as a fallback — e.g. LXC 114 (gitea) as a break-glass for the in-cluster Gitea per #187 — in which case hold #232 until you're ready. Also: the workstation `pass`/env has no Hetzner S3 backend creds, so the `tofu state rm`/apply needs to run where those creds live regardless.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#199