- ARCHITECTURE.md: System architecture documentation with ASCII diagrams
covering network topology, service architecture, HA clusters, and data flow
- OPERATIONS.md: Comprehensive operations manual with daily/weekly/monthly
tasks, troubleshooting guides, and maintenance procedures
Resolves DOC-1 from infrastructure review.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Created comprehensive chaos testing playbook that validates:
- DNS cluster failover (stops master, verifies VIP migration)
- PostgreSQL HA failover (stops keepalived, verifies VIP migration)
- RustFS S3 health check (non-destructive endpoint test)
Features:
- Records before/after state for each test
- Automatic service restoration after tests
- Test results summary with pass/fail status
- Results saved to file for historical tracking
- Tag-based execution for individual component testing
Usage:
ansible-playbook -i inventory/ plays/chaos-testing.yml
ansible-playbook -i inventory/ plays/chaos-testing.yml --tags dns
WARNING: Run during maintenance windows as it disrupts services.
Resolves OPS-3 from infrastructure review.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Added vault_grafana_authentik_client_id and
vault_grafana_authentik_client_secret for Grafana SSO.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Created grafana-authentik role to configure Grafana with Authentik SSO:
- OAuth2/OpenID Connect authentication via generic_oauth provider
- Role mapping from Authentik groups (Grafana Admins/Editors)
- PKCE support for enhanced security
- Auto sign-up for new users
Prerequisites:
1. Create OAuth2 provider in Authentik with slug "grafana"
2. Set redirect URI: https://dash.renner.uno/login/generic_oauth
3. Add credentials to vault:
- vault_grafana_authentik_client_id
- vault_grafana_authentik_client_secret
Usage: ansible-playbook -i inventory/ plays/configure-grafana-authentik.yml
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Wrapped alert-rules.yaml.j2 in {% raw %}{% endraw %} to prevent Jinja2
from interpreting Grafana's $labels templating syntax.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Added comprehensive alert rules for critical infrastructure monitoring:
Critical alerts (2min trigger):
- DNS VIP (10.2.40.53) health check
- RustFS S3 backend availability
- Proxmox node availability
- Authentik authentication service
- Caddy ingress reverse proxy
Warning alerts:
- Disk space (80% warning, 90% critical)
- PostgreSQL replication lag (>60s)
- High memory usage (>90%)
- High CPU usage (>90% for 15min)
- Generic service target down
Alerts are provisioned via Grafana provisioning system.
Run: ansible-playbook plays/configure-grafana-observability.yml
Resolves OPS-1 from infrastructure review.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Marked the following issues as resolved:
- CRITICAL-1: Vault password (env variable via vault-password.sh)
- CRITICAL-2: TLS verification (defaults to secure)
- CRITICAL-3: Automated backups (3-tier PBS strategy)
- CRITICAL-4: Authentik DB failover (keepalived VIP)
- ARCH-1: RustFS tags (marked as critical)
- OPS-2: DR runbooks (DISASTER_RECOVERY.md created)
Updated overall health status to EXCELLENT.
Only OPS-1 (monitoring alerts) remains as high priority.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Created comprehensive DISASTER_RECOVERY.md with:
- Recovery procedures for DNS cluster, RustFS S3, PostgreSQL HA, and Proxmox
- RTO/RPO targets for each component
- Diagnosis commands and step-by-step recovery instructions
- Testing schedule for quarterly DR drills
- Post-incident review template
Resolves OPS-2 from infrastructure review.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Changed authentik-db.renner.uno DNS record from CNAME pointing to
primary database node to A record pointing to keepalived VIP (10.2.40.135).
This enables automatic database failover without manual DNS changes.
When the primary PostgreSQL node fails, keepalived moves the VIP to
the standby node automatically.
Resolves CRITICAL-4 from infrastructure review.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Inventory and infrastructure fixes:
- Use FQDNs for authentik hosts in inventory (authentik-db-1.srv.renner.uno)
- Fix postgres_primary_host reference in install-authentik-ha playbook
- Disable systemd security hardening for Alloy in LXC containers
(namespace restrictions prevent these features)
- Update vault with new monitoring credentials
- Document configuration-via-ansible requirement in CLAUDE.md
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Enable mobile document scanning workflow by adding Syncthing
container to Paperless stack. Documents synced from phones
automatically appear in the consume directory.
Configuration includes:
- Optional syncthing container in docker-compose
- Automated device pairing via entrypoint script
- Pre-configured device IDs for trusted devices
- Sync folder mapped to Paperless consume directory
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Enable metrics collection for monitoring:
- Authentik: Expose metrics on port 9300
- Caddy: Enable admin API on :2019 with metrics endpoint
- Gitea: Enable built-in metrics endpoint
- Immich: Enable telemetry metrics on port 8081
These endpoints allow Prometheus to scrape application-level
metrics for the comprehensive monitoring dashboards.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add roles for collecting metrics across the infrastructure:
- pve-exporter: Proxmox VE cluster metrics
- unifi-poller: UniFi network controller metrics
- postgres-exporter: PostgreSQL database metrics
- blackbox-exporter: HTTP/TCP/DNS endpoint probing
- node-exporter: System-level metrics for all hosts
Also includes:
- Updated prometheus.yml with scrape configs for all new targets
- File-based service discovery for node exporters
- New playbooks for deploying blackbox and node exporters
- PostgreSQL monitoring user setup playbook
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The Grafana Alloy zip extracts to alloy-linux-amd64 file directly,
not a directory structure. Updated find task to locate the binary
file correctly.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Deploy observability stack with:
- Loki (LXC 226) for log aggregation with 90-day retention
- Prometheus (LXC 227) for metrics collection with 90-day retention
- Grafana Alloy monitoring agent auto-deployed to all hosts
- Grafana dashboards and alerting configuration
Ansible roles created:
- loki-server: Installs and configures Loki
- prometheus-server: Installs and configures Prometheus
- monitoring-agent: Deploys Alloy agent to all hosts
- grafana-observability: Configures Grafana data sources and dashboards
The monitoring-agent role is added to common.yml for automatic
deployment to all new and existing services.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Document opentofu-bootstrap directory structure and explain the local
backend requirement for avoiding circular dependency with RustFS.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add /etc/hosts fix to common role for all LXC containers
(Proxmox sets 127.0.1.1 which breaks services resolving own hostname)
- Add network_interface to ignore_changes in LXC module
(Prevents network disruption during tofu apply)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Pre-flight validation of RustFS availability before tofu init to fail
fast with clear error messages when S3 backend is unreachable or
degraded.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Move RustFS cluster management to a separate OpenTofu project to break
the circular dependency (OpenTofu state stored in RustFS, which was
managed by OpenTofu).
Changes:
- Create opentofu-bootstrap/ with dedicated RustFS management
- Use local state for bootstrap project (cannot use S3 - that's circular)
- Remove RustFS from main opentofu/ project
- Re-enable S3 backend in main project
- Add comprehensive README for bootstrap project
The bootstrap project manages Layer 0 infrastructure that must exist
before the main OpenTofu project can initialize.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add features to ignore_changes in LXC module (API token limitations)
- Add device_passthrough support for GPU/device passthrough in LXC
- Normalize MAC addresses to uppercase for Proxmox compatibility
- Add architecture field to CPU block
- Add keyctl to features block
- Fix tag parsing to handle both comma and semicolon separators
- Add agent_trim and USB mapping support to VM module
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Replace Telmate/proxmox v3.0.2 with bpg/proxmox v0.95.0
- Convert modules/proxmox_vm to BPG resource syntax
- Convert modules/proxmox_lxc to BPG resource syntax
- Update vm_605_template.tf to use proxmox_virtual_environment_vm
- Fix API token format in providers.tf (remove duplicate @pam)
- Add comprehensive lifecycle ignore_changes to prevent unnecessary replacements
- Handle disk size parsing for both G and M suffixes
- Fix empty tags handling with compact()
- Temporarily switch to local backend (S3 backend has quorum issues)
Breaking changes:
- Resource types changed from proxmox_lxc/proxmox_vm_qemu to
proxmox_virtual_environment_container/proxmox_virtual_environment_vm
- State was recreated by importing all resources from Proxmox cluster
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Added REVIEW_SUMMARY.md to provide an executive summary of the infrastructure review, highlighting critical issues and next steps.
- Created VAULT_PASSWORD_USAGE.md as a guide for securely managing the Ansible vault password using environment variables and password managers.
- Introduced vault-password.sh script to read the vault password from the ANSIBLE_VAULT_PASSWORD environment variable, ensuring secure access during Ansible operations.
- Removed the plaintext vault password file (.vault_pass) to enhance security.
- Added Ansible playbook for installing Proxmox DNS server in HA configuration.
- Created role for Keepalived with VIP management and health checks.
- Implemented notification script for state changes in Keepalived.
- Developed health check script to monitor DNS service status.
- Configured Keepalived with VRRP settings and health check scripts.
- Documented HA architecture and deployment process for Proxmox DNS server.
- Migrated DHCP management from UniFi to Proxmox DNS server for dynamic IP resolution.
- Removed UniFi client resources from OpenTofu modules to streamline management.
- Added scripts for managing UniFi client resources during migration.
- Created unifi_targets.txt and unifi_targets_strict.txt files to define DNS records for services including authentik, caddy, and others.
- Included ingress CNAMEs for modules such as audiobookshelf, gitea, grafana, and more.
- Ensured both files contain comprehensive lists of DNS records to support service discovery and routing.
k.rennerandClaude