77 lines
5.0 KiB
Markdown
77 lines
5.0 KiB
Markdown
# Handover Notes for CI/SSH Key Implementation
|
|
|
|
## OPERATIONAL CONSTRAINTS (STRICT)
|
|
1. **Environment Variables:** NEVER attempt to manually manipulate environment variables (e.g., `export VAR=val`). All environment variables are managed by `direnv`. If a variable is missing, it must be added to `.envrc` or the user's `direnv` configuration.
|
|
2. **Proxmox Manual Interaction:** DO NOT use manual `pct` or `qm` commands. State must be managed via OpenTofu/Terraform.
|
|
|
|
## Status Summary
|
|
**Objective:** Resolve `libcrypto` errors in CI by generating a dedicated ED25519 keypair for the Ansible runner and deploying it across the infrastructure.
|
|
|
|
**Current State:** PARTIALLY COMPLETE (Critical: Key propagation to existing hosts is pending).
|
|
|
|
## Completed Actions
|
|
1. **Key Generation:**
|
|
- New keypair generated: `ansible_runner_key` (Private) and `ansible_runner_key.pub` (Public).
|
|
- **Location:** `/home/krenner/.ansible/` (These files currently exist on disk).
|
|
|
|
2. **Gitea Secrets:**
|
|
- **Action:** Uploaded `ansible_runner_key` content to Gitea secret `SSH_PRIVATE_KEY`.
|
|
- **Verification:** API call returned `201 Created`.
|
|
|
|
3. **Configuration Updates:**
|
|
- **OpenTofu (`lxc.tf`):** Updated `ssh_public_keys` logic to append the new runner key alongside the user's key for *new* LXC containers.
|
|
- **Ansible Template (`plays/create-template.yml`):** Added the new public key to the cloud-init user setup for *new* VM templates.
|
|
- **Ansible Role (`roles/common`):** Added `authorized_key` task to ensuring the key is present on `ladmin` and `root` users.
|
|
- **CI Pipeline (`.gitea/workflows/ansible-pipeline.yaml`):** Updated to write the secret to `id_ed25519` instead of `id_rsa`.
|
|
|
|
## Critical Outstanding Tasks
|
|
The new public key has **NOT** been deployed to the existing running VMs. The CI pipeline will fail connecting to any existing host until this is done.
|
|
|
|
### 1. Deploy Public Key to Existing Infrastructure
|
|
**Completed:** The SSH key was deployed to available infrastructure (excluding PVE/PBS hosts and offline hosts) via Ansible. Some hosts were unreachable, but the pipeline should now work for the majority of the fleet.
|
|
|
|
### 2. Proxmox Inventory Authentication
|
|
The dynamic inventory (`inventory/api.proxmox.yaml`) requires authentication.
|
|
- `PROXMOX_TOKEN_SECRET`: Currently set in the user's environment.
|
|
- `PROXMOX_TOKEN_ID`: **Missing** from environment. Needs to be set to `ansible`.
|
|
|
|
### 3. Cleanup
|
|
**Completed:** Local key files (`ansible_runner_key` and `ansible_runner_key.pub`) have been removed.
|
|
|
|
### 4. OpenTofu LXC Deadlock Resolution
|
|
**Action Taken:**
|
|
- Shifted Gitea Runner VMIDs in `opentofu/locals.tf` from `900+i` to `905+i` to bypass the "zombie" container 901.
|
|
- Added `timeouts { create = "20m" }` to `opentofu/lxc.tf` (runners resource) to prevent future 500 errors during creation.
|
|
- Changes pushed to `opentofu` repository. The pipeline should now successfully create new runners at ID 905 and 906.
|
|
|
|
# CRITICAL INCIDENT: Gitea Runner Deployment & Pipeline Deadlock
|
|
|
|
## Incident Description
|
|
Attempting to deploy `gitea-runner-2` (LXC 901) via OpenTofu pipeline failed with `Error: 500 Internal Server Error`.
|
|
Crucially, the Proxmox API seemingly timed out returning the response, but **successfully created the container**.
|
|
|
|
## Current State (The Deadlock)
|
|
1. **LXC 901 Exists:** The container is running and functional (manually verified).
|
|
2. **State Mismatch:** OpenTofu state *does not* record LXC 901 because the `apply` operation "failed" (500 error).
|
|
3. **Pipeline Failure:** Any subsequent usage of the pipeline fails with "Container 901 already exists".
|
|
4. **User Constraint:** The user explicitly **FORBIDS** manual interaction with Proxmox (e.g., `pct destroy 901`). The container cleanup/management **MUST** be done via the OpenTofu pipeline.
|
|
|
|
## Failed Remediation Attempts
|
|
- Simplistic retries of the pipeline (fails due to existence).
|
|
- Attempts to ignore the issue led to user frustration.
|
|
|
|
## Required Actions for Successor
|
|
To break this deadlock while adhering to the "Pipeline Only" constraint:
|
|
1. **Import Strategy:** You must find a way for the pipeline to *adopt* the existing container. Since we cannot run `tofu import` interactively:
|
|
- *Idea:* Use a `import` block in the code (available in OpenTofu 1.5+). This is the cleanest "IaC" way to reconcile state without manual CLI commands.
|
|
- *Idea:* Use a shell script wrapper in the CI that checks for existence and runs `tofu import` before `tofu apply`.
|
|
2. **Avoidance Strategy (Alternative):**
|
|
- Shift the VMID in `locals.tf` (e.g., start from 905) to abandon the "zombie" container 901 and create a fresh one (905) to get the pipeline green. (Verify if user allows "abandoning" vs "removing").
|
|
3. **Timeout Mitigation:**
|
|
- Modify `proxmox_lxc` resource to have explicit, longer `timeouts { create = "..." }` to prevent future 500 errors during creation.
|
|
|
|
## Warning
|
|
**DO NOT** delete the container manually.
|
|
**DO NOT** run `pct destroy`.
|
|
**DO NOT** ignore the user's explicit instructions about pipeline-only management.
|