# Handover Notes for CI/SSH Key Implementation

## OPERATIONAL CONSTRAINTS (STRICT)
1.  **Environment Variables:** NEVER attempt to manually manipulate environment variables (e.g., `export VAR=val`). All environment variables are managed by `direnv`. If a variable is missing, it must be added to `.envrc` or the user's `direnv` configuration.
2.  **Proxmox Manual Interaction:** DO NOT use manual `pct` or `qm` commands. State must be managed via OpenTofu/Terraform.

## Status Summary
**Objective:** Resolve `libcrypto` errors in CI by generating a dedicated ED25519 keypair for the Ansible runner and deploying it across the infrastructure.

**Current State:** PARTIALLY COMPLETE (Critical: Key propagation to existing hosts is pending).

## Completed Actions
1.  **Key Generation:**
    -   New keypair generated: `ansible_runner_key` (Private) and `ansible_runner_key.pub` (Public).
    -   **Location:** `/home/krenner/.ansible/` (These files currently exist on disk).

2.  **Gitea Secrets:**
    -   **Action:** Uploaded `ansible_runner_key` content to Gitea secret `SSH_PRIVATE_KEY`.
    -   **Verification:** API call returned `201 Created`.

3.  **Configuration Updates:**
    -   **OpenTofu (`lxc.tf`):** Updated `ssh_public_keys` logic to append the new runner key alongside the user's key for *new* LXC containers.
    -   **Ansible Template (`plays/create-template.yml`):** Added the new public key to the cloud-init user setup for *new* VM templates.
    -   **Ansible Role (`roles/common`):** Added `authorized_key` task to ensuring the key is present on `ladmin` and `root` users.
    -   **CI Pipeline (`.gitea/workflows/ansible-pipeline.yaml`):** Updated to write the secret to `id_ed25519` instead of `id_rsa`.

## Critical Outstanding Tasks
The new public key has **NOT** been deployed to the existing running VMs. The CI pipeline will fail connecting to any existing host until this is done.

### 1. Deploy Public Key to Existing Infrastructure
**Completed:** The SSH key was deployed to available infrastructure (excluding PVE/PBS hosts and offline hosts) via Ansible. Some hosts were unreachable, but the pipeline should now work for the majority of the fleet.

### 2. Proxmox Inventory Authentication
The dynamic inventory (`inventory/api.proxmox.yaml`) requires authentication.
-   `PROXMOX_TOKEN_SECRET`: Currently set in the user's environment.
-   `PROXMOX_TOKEN_ID`: **Missing** from environment. Needs to be set to `ansible`.

### 3. Cleanup
**Completed:** Local key files (`ansible_runner_key` and `ansible_runner_key.pub`) have been removed.

### 4. OpenTofu LXC Deadlock Resolution
**Action Taken:**
-   Shifted Gitea Runner VMIDs in `opentofu/locals.tf` from `900+i` to `905+i` to bypass the "zombie" container 901.
-   Added `timeouts { create = "20m" }` to `opentofu/lxc.tf` (runners resource) to prevent future 500 errors during creation.
-   Changes pushed to `opentofu` repository. The pipeline should now successfully create new runners at ID 905 and 906.

# CRITICAL INCIDENT: Gitea Runner Deployment & Pipeline Deadlock

## Incident Description
Attempting to deploy `gitea-runner-2` (LXC 901) via OpenTofu pipeline failed with `Error: 500 Internal Server Error`.
Crucially, the Proxmox API seemingly timed out returning the response, but **successfully created the container**.

## Current State (The Deadlock)
1.  **LXC 901 Exists:** The container is running and functional (manually verified).
2.  **State Mismatch:** OpenTofu state *does not* record LXC 901 because the `apply` operation "failed" (500 error).
3.  **Pipeline Failure:** Any subsequent usage of the pipeline fails with "Container 901 already exists".
4.  **User Constraint:** The user explicitly **FORBIDS** manual interaction with Proxmox (e.g., `pct destroy 901`). The container cleanup/management **MUST** be done via the OpenTofu pipeline.

## Failed Remediation Attempts
-   Simplistic retries of the pipeline (fails due to existence).
-   Attempts to ignore the issue led to user frustration.

## Required Actions for Successor
To break this deadlock while adhering to the "Pipeline Only" constraint:
1.  **Import Strategy:** You must find a way for the pipeline to *adopt* the existing container. Since we cannot run `tofu import` interactively:
    -   *Idea:* Use a `import` block in the code (available in OpenTofu 1.5+). This is the cleanest "IaC" way to reconcile state without manual CLI commands.
    -   *Idea:* Use a shell script wrapper in the CI that checks for existence and runs `tofu import` before `tofu apply`.
2.  **Avoidance Strategy (Alternative):**
    -   Shift the VMID in `locals.tf` (e.g., start from 905) to abandon the "zombie" container 901 and create a fresh one (905) to get the pipeline green. (Verify if user allows "abandoning" vs "removing").
3.  **Timeout Mitigation:**
    -   Modify `proxmox_lxc` resource to have explicit, longer `timeouts { create = "..." }` to prevent future 500 errors during creation.

## Warning
**DO NOT** delete the container manually.
**DO NOT** run `pct destroy`.
**DO NOT** ignore the user's explicit instructions about pipeline-only management.