k.rennerandClaude
132be92281
- ARCHITECTURE.md: System architecture documentation with ASCII diagrams covering network topology, service architecture, HA clusters, and data flow - OPERATIONS.md: Comprehensive operations manual with daily/weekly/monthly tasks, troubleshooting guides, and maintenance procedures Resolves DOC-1 from infrastructure review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
519 lines
27 KiB
Markdown
519 lines
27 KiB
Markdown
# Infrastructure Architecture
|
|
|
|
**Last Updated:** 2026-02-18
|
|
**Version:** 1.0
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
This homelab infrastructure runs on a 3-node Proxmox VE cluster managing 40+ services through Infrastructure as Code (OpenTofu + Ansible). The architecture emphasizes high availability, automated provisioning, and security through SSO.
|
|
|
|
---
|
|
|
|
## Physical Layer
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ PROXMOX VE CLUSTER │
|
|
│ │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ JUPITER │ │ SATURN │ │ NEPTUNE │ │
|
|
│ │ 10.2.40.201 │ │ 10.2.40.202 │ │ 10.2.40.203 │ │
|
|
│ │ │ │ │ │ │ │
|
|
│ │ - VMs │ │ - VMs │ │ - VMs │ │
|
|
│ │ - LXC │ │ - LXC │ │ - LXC │ │
|
|
│ │ - ZFS │ │ - ZFS │ │ - ZFS │ │
|
|
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
|
|
│ │ │ │ │
|
|
│ └───────────────────┼───────────────────┘ │
|
|
│ │ │
|
|
│ Corosync Cluster │
|
|
│ (HA + Live Migration) │
|
|
└─────────────────────────────┼────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────┐
|
|
│ UniFi Gateway │
|
|
│ 10.2.40.1 │
|
|
└────────┬────────┘
|
|
│
|
|
▼
|
|
Internet
|
|
```
|
|
|
|
### Hardware Specifications
|
|
|
|
| Node | Role | CPU | RAM | Storage |
|
|
|------|------|-----|-----|---------|
|
|
| jupiter | Primary | - | - | local-zfs |
|
|
| saturn | Secondary | - | - | local-zfs |
|
|
| neptune | Secondary | - | - | local-zfs |
|
|
|
|
---
|
|
|
|
## Network Architecture
|
|
|
|
### VLAN Layout
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ NETWORK VLANS │
|
|
├─────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ VLAN 40 - Services (v40srv) 10.2.40.0/24 │
|
|
│ ├── Infrastructure: DNS, RustFS, Caddy, Authentik │
|
|
│ ├── Monitoring: Grafana, InfluxDB, Prometheus, Loki │
|
|
│ ├── Applications: Gitea, Immich, Paperless, Home Assistant │
|
|
│ └── Databases: PostgreSQL HA cluster │
|
|
│ │
|
|
│ VLAN 45 - Ingress/DMZ (v42ing) 10.2.45.0/24 │
|
|
│ └── Public-facing services via Cloudflare Tunnel │
|
|
│ │
|
|
│ VLAN 50 - Private (v50pri) 10.2.50.0/24 │
|
|
│ └── Workstations, development machines │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### IP Allocation
|
|
|
|
#### VIP Addresses (10.2.40.50-99)
|
|
| IP | Service | Managed By |
|
|
|----|---------|------------|
|
|
| 10.2.40.53 | DNS Cluster VIP | keepalived |
|
|
| 10.2.40.101 | RustFS S3 VIP | keepalived |
|
|
| 10.2.40.135 | PostgreSQL HA VIP | keepalived |
|
|
|
|
#### Static Infrastructure (10.2.40.1-49)
|
|
| IP | Service |
|
|
|----|---------|
|
|
| 10.2.40.1 | UniFi Gateway |
|
|
| 10.2.40.17 | Grafana |
|
|
| 10.2.40.54-56 | DNS Cluster nodes |
|
|
|
|
#### Proxmox Nodes (10.2.40.201-203)
|
|
| IP | Node |
|
|
|----|------|
|
|
| 10.2.40.201 | jupiter |
|
|
| 10.2.40.202 | neptune |
|
|
| 10.2.40.203 | saturn |
|
|
|
|
---
|
|
|
|
## Service Architecture
|
|
|
|
### Traffic Flow
|
|
|
|
```
|
|
Internet
|
|
│
|
|
▼
|
|
┌───────────────────┐
|
|
│ Cloudflare │
|
|
│ (DNS + WAF) │
|
|
└─────────┬─────────┘
|
|
│
|
|
▼
|
|
┌───────────────────┐
|
|
│ Cloudflared │
|
|
│ Tunnel (LXC) │
|
|
└─────────┬─────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ CADDY HA CLUSTER │
|
|
│ (Reverse Proxy + TLS) │
|
|
│ │
|
|
│ caddy-1.srv.renner.uno ◄──keepalived──► caddy-2.srv.renner.uno │
|
|
│ VIP │
|
|
└─────────────────────────────────┬───────────────────────────────────────┘
|
|
│
|
|
┌─────────────┼─────────────┐
|
|
│ │ │
|
|
▼ ▼ ▼
|
|
┌───────────┐ ┌───────────┐ ┌───────────┐
|
|
│ Authentik │ │ Grafana │ │ Gitea │
|
|
│ (SSO) │ │ (Dash) │ │ (Git) │
|
|
└─────┬─────┘ └───────────┘ └───────────┘
|
|
│
|
|
▼
|
|
┌─────────────────┐
|
|
│ All Protected │
|
|
│ Applications │
|
|
└─────────────────┘
|
|
```
|
|
|
|
### High Availability Components
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ HIGH AVAILABILITY DESIGN │
|
|
├─────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ DNS CLUSTER (3 nodes) │
|
|
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
|
|
│ │ DNS-1 │ │ DNS-2 │ │ DNS-3 │ │
|
|
│ │ MASTER │ │ BACKUP │ │ BACKUP │ │
|
|
│ │ .54 │ │ .55 │ │ .56 │ │
|
|
│ └────┬────┘ └────┬────┘ └────┬────┘ │
|
|
│ └─────────────┼─────────────┘ │
|
|
│ ▼ │
|
|
│ VIP: 10.2.40.53 │
|
|
│ Failover: 3-5 seconds │
|
|
│ │
|
|
│ ───────────────────────────────────────────────────────────────────── │
|
|
│ │
|
|
│ POSTGRESQL HA (2 nodes + streaming replication) │
|
|
│ ┌─────────────────┐ ┌─────────────────┐ │
|
|
│ │ authentik-db-1 │──────►│ authentik-db-2 │ │
|
|
│ │ PRIMARY │ repl │ STANDBY │ │
|
|
│ └────────┬────────┘ └────────┬────────┘ │
|
|
│ └────────────┬────────────┘ │
|
|
│ ▼ │
|
|
│ VIP: 10.2.40.135 │
|
|
│ Failover: automatic │
|
|
│ │
|
|
│ ───────────────────────────────────────────────────────────────────── │
|
|
│ │
|
|
│ RUSTFS S3 CLUSTER (3 nodes) │
|
|
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
|
|
│ │ RFS-1 │ │ RFS-2 │ │ RFS-3 │ │
|
|
│ │ .120 │ │ .167 │ │ .218 │ │
|
|
│ └────┬────┘ └────┬────┘ └────┬────┘ │
|
|
│ └─────────────┼─────────────┘ │
|
|
│ ▼ │
|
|
│ VIP: s3.renner.uno │
|
|
│ Purpose: OpenTofu state backend │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Service Inventory
|
|
|
|
### Core Infrastructure
|
|
|
|
| Service | Type | VMID | Node | Purpose |
|
|
|---------|------|------|------|---------|
|
|
| proxmox-dns-1 | LXC | 118 | jupiter | DNS resolution (MASTER) |
|
|
| proxmox-dns-2 | LXC | 218 | saturn | DNS resolution (BACKUP) |
|
|
| proxmox-dns-3 | LXC | 318 | neptune | DNS resolution (BACKUP) |
|
|
| rustfs-1/2/3 | LXC | 800+ | distributed | S3 state backend |
|
|
| cloudflared | LXC | 117 | - | Cloudflare tunnel |
|
|
|
|
### Authentication & Identity
|
|
|
|
| Service | Type | VMID | Purpose |
|
|
|---------|------|------|---------|
|
|
| authentik-app-1 | LXC | 130 | Identity provider |
|
|
| authentik-app-2 | LXC | 131 | Identity provider (HA) |
|
|
| authentik-db-1 | LXC | 132 | PostgreSQL primary |
|
|
| authentik-db-2 | LXC | 133 | PostgreSQL standby |
|
|
|
|
### Monitoring Stack
|
|
|
|
| Service | Type | VMID | Purpose |
|
|
|---------|------|------|---------|
|
|
| influxdb | VM | 108 | Time-series database |
|
|
| grafana | VM | 109 | Visualization (dash.renner.uno) |
|
|
| prometheus | LXC | - | Metrics collection |
|
|
| loki | LXC | - | Log aggregation |
|
|
| wazuh | LXC | 222 | Security monitoring |
|
|
|
|
### Applications
|
|
|
|
| Service | Type | VMID | Purpose |
|
|
|---------|------|------|---------|
|
|
| gitea | LXC | 114 | Git server + CI/CD |
|
|
| immich | LXC | 225 | Photo management |
|
|
| paperless | LXC | - | Document management |
|
|
| home-assistant | VM | 103 | Home automation |
|
|
|
|
### CI/CD
|
|
|
|
| Service | Type | VMID | Purpose |
|
|
|---------|------|------|---------|
|
|
| gitea-runner-1 | LXC | 905 | CI/CD execution |
|
|
| gitea-runner-2 | LXC | 906 | CI/CD execution |
|
|
|
|
---
|
|
|
|
## Data Flow
|
|
|
|
### DNS Resolution
|
|
|
|
```
|
|
Client Query: gitea.srv.renner.uno
|
|
│
|
|
▼
|
|
┌───────────────────┐
|
|
│ DNS VIP │
|
|
│ 10.2.40.53 │
|
|
└─────────┬─────────┘
|
|
│
|
|
▼
|
|
┌───────────────────┐
|
|
│ Proxmox DNS │
|
|
│ (Custom Server) │
|
|
│ │
|
|
│ 1. Check cache │
|
|
│ 2. Query PVE API │
|
|
│ 3. Return IP │
|
|
└─────────┬─────────┘
|
|
│
|
|
▼
|
|
┌───────────────────┐
|
|
│ Response: │
|
|
│ 10.2.40.x │
|
|
└───────────────────┘
|
|
```
|
|
|
|
### Authentication Flow
|
|
|
|
```
|
|
User: https://grafana.renner.uno
|
|
│
|
|
▼
|
|
┌───────────────────┐
|
|
│ Caddy Ingress │
|
|
│ (TLS termination)│
|
|
└─────────┬─────────┘
|
|
│
|
|
▼
|
|
┌───────────────────┐
|
|
│ Grafana │
|
|
│ "Login with │
|
|
│ Authentik" │
|
|
└─────────┬─────────┘
|
|
│
|
|
▼
|
|
┌───────────────────┐
|
|
│ Authentik │
|
|
│ (auth.renner.uno)│
|
|
│ │
|
|
│ - OIDC/OAuth2 │
|
|
│ - MFA (TOTP) │
|
|
│ - Group mapping │
|
|
└─────────┬─────────┘
|
|
│
|
|
▼
|
|
┌───────────────────┐
|
|
│ Authenticated │
|
|
│ Session Created │
|
|
└───────────────────┘
|
|
```
|
|
|
|
### Infrastructure Provisioning
|
|
|
|
```
|
|
Developer: git push
|
|
│
|
|
▼
|
|
┌───────────────────┐
|
|
│ Gitea │
|
|
│ (Trigger CI) │
|
|
└─────────┬─────────┘
|
|
│
|
|
▼
|
|
┌───────────────────┐
|
|
│ Gitea Runner │
|
|
│ │
|
|
│ 1. tofu init │
|
|
│ 2. tofu plan │
|
|
│ 3. tofu apply │
|
|
└─────────┬─────────┘
|
|
│
|
|
▼
|
|
┌───────────────────┐ ┌───────────────────┐
|
|
│ RustFS S3 │◄─────►│ Proxmox API │
|
|
│ (State Backend) │ │ (Create VM/LXC) │
|
|
└───────────────────┘ └───────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Dependency Map
|
|
|
|
```
|
|
┌─────────────┐
|
|
│ Internet │
|
|
└──────┬──────┘
|
|
│
|
|
┌──────▼──────┐
|
|
│ Cloudflare │
|
|
└──────┬──────┘
|
|
│
|
|
┌──────▼──────┐
|
|
│ Cloudflared │
|
|
└──────┬──────┘
|
|
│
|
|
┌───────────────────────────┼───────────────────────────┐
|
|
│ │ │
|
|
▼ ▼ ▼
|
|
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
|
│ Caddy Ingress │ │ DNS Cluster │◄────────│ Proxmox API │
|
|
│ (All traffic) │ │ (Name resolution) │ (Discovery) │
|
|
└────────┬────────┘ └────────┬────────┘ └─────────────────┘
|
|
│ │
|
|
│ │
|
|
▼ │
|
|
┌─────────────────┐ │
|
|
│ Authentik │◄─────────────────┘
|
|
│ (SSO/Identity)│
|
|
└────────┬────────┘
|
|
│
|
|
│ Protects
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ PROTECTED SERVICES │
|
|
│ │
|
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
|
│ │ Grafana │ │ Gitea │ │ Immich │ │Paperless │ │ ... │ │
|
|
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Critical Dependencies
|
|
|
|
| Service | Depends On | Impact if Down |
|
|
|---------|------------|----------------|
|
|
| All services | DNS Cluster | No name resolution |
|
|
| All services | Caddy Ingress | No external access |
|
|
| Protected apps | Authentik | No authentication |
|
|
| OpenTofu | RustFS S3 | Cannot provision |
|
|
| Authentik | PostgreSQL HA | Auth database unavailable |
|
|
|
|
---
|
|
|
|
## Backup Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 3-TIER BACKUP STRATEGY │
|
|
├─────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ TIER 1: Proxmox VE Cluster (Source) │
|
|
│ ├── All VMs and LXC containers │
|
|
│ ├── Automated snapshot creation │
|
|
│ └── QEMU guest agent for consistency │
|
|
│ │ │
|
|
│ ▼ Automatic backup │
|
|
│ │
|
|
│ TIER 2: PBS2 (Primary Backup) │
|
|
│ ├── Deduplication + compression │
|
|
│ ├── Retention: 7 daily, 4 weekly, 12 monthly │
|
|
│ └── Separate NAS infrastructure │
|
|
│ │ │
|
|
│ ▼ Automatic replication │
|
|
│ │
|
|
│ TIER 3: pbs.obg (Offsite) │
|
|
│ ├── Geographic separation │
|
|
│ ├── Air-gapped from primary │
|
|
│ └── Ransomware protection │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Security Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ SECURITY LAYERS │
|
|
├─────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ LAYER 1: Network Perimeter │
|
|
│ ├── Cloudflare WAF + DDoS protection │
|
|
│ ├── UniFi firewall rules │
|
|
│ └── VLAN segmentation │
|
|
│ │
|
|
│ LAYER 2: Application Security │
|
|
│ ├── TLS everywhere (Let's Encrypt via Caddy) │
|
|
│ ├── Authentik SSO with MFA │
|
|
│ └── OAuth2/OIDC for service integration │
|
|
│ │
|
|
│ LAYER 3: Infrastructure Security │
|
|
│ ├── SSH key-based authentication only │
|
|
│ ├── Ansible Vault for secrets │
|
|
│ └── API tokens with minimal permissions │
|
|
│ │
|
|
│ LAYER 4: Monitoring & Detection │
|
|
│ ├── Wazuh SIEM │
|
|
│ ├── Trivy vulnerability scanning (weekly) │
|
|
│ └── Grafana alerting │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Technology Stack
|
|
|
|
### Infrastructure as Code
|
|
|
|
| Tool | Purpose | State Storage |
|
|
|------|---------|---------------|
|
|
| OpenTofu | Infrastructure provisioning | RustFS S3 |
|
|
| Ansible | Configuration management | Git (vault encrypted) |
|
|
| Gitea Actions | CI/CD automation | - |
|
|
|
|
### Providers & Integrations
|
|
|
|
| Provider | Purpose |
|
|
|----------|---------|
|
|
| bpg/proxmox | VM/LXC management |
|
|
| ubiquiti-community/unifi | DNS records |
|
|
| hashicorp/aws | S3 backend access |
|
|
|
|
### Runtime
|
|
|
|
| Component | Technology |
|
|
|-----------|------------|
|
|
| Containers | Proxmox LXC (Debian) |
|
|
| VMs | Proxmox QEMU/KVM |
|
|
| Orchestration | Proxmox HA + keepalived |
|
|
| Storage | ZFS (local-zfs) |
|
|
|
|
---
|
|
|
|
## Scaling Considerations
|
|
|
|
### Current Capacity
|
|
- 3 Proxmox nodes
|
|
- 40+ services
|
|
- Adequate for homelab use
|
|
|
|
### Growth Paths
|
|
|
|
1. **Add Proxmox Node**
|
|
- Join to existing cluster
|
|
- Automatic HA rebalancing
|
|
- Update OpenTofu node targets
|
|
|
|
2. **Add Storage**
|
|
- Expand ZFS pools
|
|
- Add Ceph for distributed storage
|
|
|
|
3. **Geographic Distribution**
|
|
- Site-to-site VPN
|
|
- Database replication across sites
|
|
- Multi-region RustFS
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
- `CLAUDE.md` - Project overview and commands
|
|
- `SECURITY.md` - Access control and procedures
|
|
- `DISASTER_RECOVERY.md` - Recovery runbooks
|
|
- `BACKUP_INFRASTRUCTURE.md` - Backup details
|
|
- `ansible/DOCUMENTATION.md` - Ansible usage
|
|
|
|
---
|
|
|
|
*Document maintained by infrastructure team. Update when architecture changes.*
|