homelab/OPERATIONS.md

Operations Manual

Last Updated: 2026-02-18 On-Call: Infrastructure Owner

---

Quick Reference

Emergency Commands

# Check all critical services
ssh root@10.2.40.54 'systemctl status proxmox-dns keepalived'
dig @10.2.40.53 gitea.srv.renner.uno
curl -s http://s3.renner.uno:9000/minio/health/live

# Restart DNS cluster (if VIP issues)
for ip in 10.2.40.54 10.2.40.55 10.2.40.56; do
  ssh root@$ip 'systemctl restart keepalived proxmox-dns'
done

# Check PostgreSQL replication
ssh root@authentik-db-1.srv.renner.uno 'sudo -u postgres psql -c "SELECT * FROM pg_stat_replication;"'

Key URLs

Service	URL	Purpose
Grafana	https://dash.renner.uno	Monitoring dashboards
Authentik	https://auth.renner.uno	Identity management
Gitea	https://gitea.renner.uno	Git + CI/CD
Proxmox	https://jupiter:8006	Cluster management

---

Daily Operations

Automated Health Checks

The following are monitored automatically via Grafana alerts:

• DNS VIP responding (10.2.40.53)
• RustFS S3 accessible
• PostgreSQL replication lag < 60s
• Disk space < 80%
• All Proxmox nodes online
• Certificate expiration > 30 days

Manual Verification (if alerts not configured)

# DNS health
dig @10.2.40.53 gitea.srv.renner.uno +short

# S3 health
curl -s http://s3.renner.uno:9000/minio/health/live && echo "OK"

# PostgreSQL status
ssh root@authentik-db-1.srv.renner.uno 'sudo -u postgres pg_isready'

# Disk space
ansible all -i inventory/ -m shell -a 'df -h / | tail -1'

---

Weekly Tasks

1. Review Security Alerts

# Check Wazuh for security events
# Access via: https://wazuh.renner.uno (if exposed)

# Review failed SSH attempts
ansible all -i inventory/ -m shell -a 'grep "Failed password" /var/log/auth.log | tail -20'

2. Verify Backup Status

# Check Proxmox backup jobs (via UI)
# Datacenter > Backup > View recent jobs

# Verify backup exists in PBS
ssh root@jupiter 'pvesh get /cluster/backup'

3. Check Trivy Security Scan

# View latest scan results in Gitea Actions
# Repository > Actions > security-scan workflow

4. Review Disk Space Trends

# Check all nodes
for node in jupiter saturn neptune; do
  echo "=== $node ==="
  ssh root@$node 'zpool list && df -h /'
done

---

Monthly Tasks

1. Chaos Testing (Failover Validation)

cd /home/krenner/homelab/ansible
ansible-playbook -i inventory/ plays/chaos-testing.yml

# Or test specific component
ansible-playbook -i inventory/ plays/chaos-testing.yml --tags dns
ansible-playbook -i inventory/ plays/chaos-testing.yml --tags postgres

2. Backup Restore Test

# 1. Select random non-critical VM/LXC
# 2. Restore to test ID (9xx range)
# 3. Verify it boots and services work
# 4. Delete test instance

# Via Proxmox UI:
# Datacenter > Storage > PBS2 > Content > Select backup > Restore

3. Update Dependencies

# Update Ansible roles
cd /home/krenner/homelab/ansible
ansible-galaxy install -r requirements.yml --force

# Check OpenTofu provider updates
cd /home/krenner/homelab/opentofu
tofu init -upgrade
tofu plan  # Review for breaking changes

4. Certificate Expiration Check

# Check all certificates via Caddy
ssh root@caddy-1.srv.renner.uno 'caddy list-certificates'

# Or check specific domain
echo | openssl s_client -servername grafana.renner.uno -connect grafana.renner.uno:443 2>/dev/null | openssl x509 -noout -dates

---

Quarterly Tasks

1. SSH Key Rotation

See SECURITY.md for full procedure:

# Generate new key
ssh-keygen -t ed25519 -C "krenner@fortress-$(date +%Y%m)" -f ~/.ssh/id_ed25519_new

# Update infrastructure
# Edit opentofu/locals.tf with new public key
cd opentofu && tofu apply

# Update CI secrets in Gitea
# Test, then remove old key after 1 week

2. Full Disaster Recovery Drill

# Follow procedures in DISASTER_RECOVERY.md
# Document results and update procedures

3. Security Audit

See SECURITY.md quarterly checklist:

• Review access control matrix
• Verify MFA enabled for all admins
• Check for orphaned accounts
• Review firewall rules

4. Capacity Planning

# Review resource utilization trends in Grafana
# Check: CPU, memory, disk, network

# Identify candidates for:
# - Right-sizing (over/under-provisioned)
# - Consolidation
# - Expansion

---

Common Procedures

Deploy Infrastructure Changes

cd /home/krenner/homelab/opentofu

# Preview changes
tofu plan

# Apply changes (interactive)
tofu apply

# Or via CI/CD:
# Push to master, CI runs plan
# Approve in Gitea for apply

Deploy Configuration Changes

cd /home/krenner/homelab/ansible

# Run specific playbook
ansible-playbook -i inventory/ plays/<playbook>.yml

# Limit to specific host
ansible-playbook -i inventory/ plays/<playbook>.yml --limit hostname

# Dry run
ansible-playbook -i inventory/ plays/<playbook>.yml --check --diff

Add New Service

1. Create OpenTofu resource:

   # Create opentofu/lxc_<id>_<name>.tf
   # Use existing files as template
   tofu plan
   tofu apply
   

2. Create Ansible role (if needed):

   mkdir -p ansible/roles/<service>/{tasks,templates,defaults,handlers}
   # Create tasks/main.yml, etc.
   

3. Configure service:

   ansible-playbook -i inventory/ plays/install-<service>.yml
   

4. Add to Authentik (if web service):

   • Create OAuth2 provider
   • Create application
   • Add credentials to vault
   • Configure service for SSO

5. Add monitoring:

   • Create Grafana dashboard
   • Add Prometheus scrape target
   • Configure alerts

Rotate Secrets

cd /home/krenner/homelab/ansible

# Edit vault
ansible-vault edit inventory/group_vars/all/vault.yml

# Re-encrypt with new password (if needed)
ansible-vault rekey inventory/group_vars/all/vault.yml

# Deploy updated secrets
ansible-playbook -i inventory/ plays/<relevant-playbook>.yml

Scale Service

# For LXC containers - update OpenTofu
# Edit opentofu/lxc_<id>_<name>.tf
# Adjust: memory, cores, disk_size

tofu plan
tofu apply

# For VMs - similar process
# May require VM shutdown for some changes

---

Troubleshooting

DNS Not Resolving

# 1. Check VIP is assigned
for ip in 10.2.40.54 10.2.40.55 10.2.40.56; do
  echo "=== $ip ==="
  ssh root@$ip 'ip addr show | grep 10.2.40.53' && echo "HAS VIP"
done

# 2. Check DNS service
ssh root@10.2.40.54 'systemctl status proxmox-dns'
ssh root@10.2.40.54 'journalctl -u proxmox-dns -n 50'

# 3. Check keepalived
ssh root@10.2.40.54 'systemctl status keepalived'
ssh root@10.2.40.54 'journalctl -u keepalived -n 50'

# 4. Restart if needed
ssh root@10.2.40.54 'systemctl restart proxmox-dns keepalived'

Authentik Login Failing

# 1. Check Authentik services
ssh root@authentik-app-1.srv.renner.uno 'docker ps'
ssh root@authentik-app-1.srv.renner.uno 'docker logs authentik-server --tail 50'

# 2. Check database connectivity
ssh root@authentik-app-1.srv.renner.uno 'docker exec authentik-server ak test_database_connection'

# 3. Check PostgreSQL
ssh root@authentik-db-1.srv.renner.uno 'sudo -u postgres psql -c "SELECT 1;"'

# 4. Check VIP
ssh root@authentik-db-1.srv.renner.uno 'ip addr show | grep 10.2.40.135'

OpenTofu State Issues

# 1. Check S3 backend
curl -s http://s3.renner.uno:9000/minio/health/live
mc ls rfs/tfstate/

# 2. If state locked
tofu force-unlock <lock-id>

# 3. If state corrupted, restore from version
mc ls --versions rfs/tfstate/terraform.tfstate
mc cp --version-id <good-version> rfs/tfstate/terraform.tfstate rfs/tfstate/terraform.tfstate

High CPU/Memory on Node

# 1. Identify resource hogs
ssh root@<node> 'htop'  # or 'top'

# 2. Check VM/LXC usage
ssh root@<node> 'qm list'  # VMs
ssh root@<node> 'pct list' # Containers

# 3. Check specific container
ssh root@<node> 'pct exec <vmid> -- top -bn1 | head -20'

# 4. Consider live migration
ssh root@<node> 'qm migrate <vmid> <target-node> --online'

Service Not Starting

# 1. Check systemd status
ssh root@<host> 'systemctl status <service>'

# 2. Check logs
ssh root@<host> 'journalctl -u <service> -n 100'

# 3. Check disk space
ssh root@<host> 'df -h'

# 4. Check memory
ssh root@<host> 'free -h'

# 5. Try restart
ssh root@<host> 'systemctl restart <service>'

---

Maintenance Windows

Planned Maintenance Procedure

1. Announce maintenance (if applicable)

2. Pre-maintenance checks:

   # Verify backups are current
   # Check cluster health
   pvesh get /cluster/status
   

3. Perform maintenance:

   • Follow specific procedure
   • Document any issues

4. Post-maintenance verification:

   # Run health checks
   ansible-playbook -i inventory/ plays/chaos-testing.yml --tags rustfs
   
   # Verify services
   curl https://dash.renner.uno
   curl https://auth.renner.uno
   

5. Document completion

Rolling Updates (Zero Downtime)

# For HA services, update one node at a time

# Example: DNS cluster
for ip in 10.2.40.54 10.2.40.55 10.2.40.56; do
  echo "=== Updating $ip ==="
  ssh root@$ip 'apt update && apt upgrade -y'
  ssh root@$ip 'systemctl restart proxmox-dns'
  sleep 30  # Wait for stabilization
  dig @10.2.40.53 test.srv.renner.uno  # Verify
done

---

Contacts & Escalation

Primary Contact

• Infrastructure Owner: krenner

External Resources

• Proxmox Forum: https://forum.proxmox.com/
• Authentik Docs: https://goauthentik.io/docs/
• OpenTofu Docs: https://opentofu.org/docs/

When to Escalate

1. Complete cluster failure - All 3 Proxmox nodes down
2. Data loss suspected - Backup restoration needed
3. Security breach - Follow incident response in SECURITY.md
4. Extended outage - Cannot resolve within 2 hours

---

Appendix: Service Restart Commands

Service	Restart Command
DNS Cluster	ssh root@10.2.40.54 'systemctl restart proxmox-dns keepalived'
Authentik	ssh root@authentik-app-1.srv.renner.uno 'docker compose restart'
Grafana	ssh ladmin@grafana.srv.renner.uno 'sudo systemctl restart grafana-server'
Gitea	ssh root@gitea.srv.renner.uno 'systemctl restart gitea'
Caddy	ssh root@caddy-1.srv.renner.uno 'systemctl restart caddy'
PostgreSQL	ssh root@authentik-db-1.srv.renner.uno 'systemctl restart postgresql'

---

Document maintained by infrastructure team. Update procedures as systems change.