k.rennerandClaude
132be92281
- ARCHITECTURE.md: System architecture documentation with ASCII diagrams covering network topology, service architecture, HA clusters, and data flow - OPERATIONS.md: Comprehensive operations manual with daily/weekly/monthly tasks, troubleshooting guides, and maintenance procedures Resolves DOC-1 from infrastructure review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
27 KiB
27 KiB
Infrastructure Architecture
Last Updated: 2026-02-18 Version: 1.0
Overview
This homelab infrastructure runs on a 3-node Proxmox VE cluster managing 40+ services through Infrastructure as Code (OpenTofu + Ansible). The architecture emphasizes high availability, automated provisioning, and security through SSO.
Physical Layer
┌─────────────────────────────────────────────────────────────────────────┐
│ PROXMOX VE CLUSTER │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ JUPITER │ │ SATURN │ │ NEPTUNE │ │
│ │ 10.2.40.201 │ │ 10.2.40.202 │ │ 10.2.40.203 │ │
│ │ │ │ │ │ │ │
│ │ - VMs │ │ - VMs │ │ - VMs │ │
│ │ - LXC │ │ - LXC │ │ - LXC │ │
│ │ - ZFS │ │ - ZFS │ │ - ZFS │ │
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
│ └───────────────────┼───────────────────┘ │
│ │ │
│ Corosync Cluster │
│ (HA + Live Migration) │
└─────────────────────────────┼────────────────────────────────────────────┘
│
▼
┌─────────────────┐
│ UniFi Gateway │
│ 10.2.40.1 │
└────────┬────────┘
│
▼
Internet
Hardware Specifications
| Node | Role | CPU | RAM | Storage |
|---|---|---|---|---|
| jupiter | Primary | - | - | local-zfs |
| saturn | Secondary | - | - | local-zfs |
| neptune | Secondary | - | - | local-zfs |
Network Architecture
VLAN Layout
┌─────────────────────────────────────────────────────────────────────────┐
│ NETWORK VLANS │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ VLAN 40 - Services (v40srv) 10.2.40.0/24 │
│ ├── Infrastructure: DNS, RustFS, Caddy, Authentik │
│ ├── Monitoring: Grafana, InfluxDB, Prometheus, Loki │
│ ├── Applications: Gitea, Immich, Paperless, Home Assistant │
│ └── Databases: PostgreSQL HA cluster │
│ │
│ VLAN 45 - Ingress/DMZ (v42ing) 10.2.45.0/24 │
│ └── Public-facing services via Cloudflare Tunnel │
│ │
│ VLAN 50 - Private (v50pri) 10.2.50.0/24 │
│ └── Workstations, development machines │
│ │
└─────────────────────────────────────────────────────────────────────────┘
IP Allocation
VIP Addresses (10.2.40.50-99)
| IP | Service | Managed By |
|---|---|---|
| 10.2.40.53 | DNS Cluster VIP | keepalived |
| 10.2.40.101 | RustFS S3 VIP | keepalived |
| 10.2.40.135 | PostgreSQL HA VIP | keepalived |
Static Infrastructure (10.2.40.1-49)
| IP | Service |
|---|---|
| 10.2.40.1 | UniFi Gateway |
| 10.2.40.17 | Grafana |
| 10.2.40.54-56 | DNS Cluster nodes |
Proxmox Nodes (10.2.40.201-203)
| IP | Node |
|---|---|
| 10.2.40.201 | jupiter |
| 10.2.40.202 | neptune |
| 10.2.40.203 | saturn |
Service Architecture
Traffic Flow
Internet
│
▼
┌───────────────────┐
│ Cloudflare │
│ (DNS + WAF) │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Cloudflared │
│ Tunnel (LXC) │
└─────────┬─────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ CADDY HA CLUSTER │
│ (Reverse Proxy + TLS) │
│ │
│ caddy-1.srv.renner.uno ◄──keepalived──► caddy-2.srv.renner.uno │
│ VIP │
└─────────────────────────────────┬───────────────────────────────────────┘
│
┌─────────────┼─────────────┐
│ │ │
▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐
│ Authentik │ │ Grafana │ │ Gitea │
│ (SSO) │ │ (Dash) │ │ (Git) │
└─────┬─────┘ └───────────┘ └───────────┘
│
▼
┌─────────────────┐
│ All Protected │
│ Applications │
└─────────────────┘
High Availability Components
┌─────────────────────────────────────────────────────────────────────────┐
│ HIGH AVAILABILITY DESIGN │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ DNS CLUSTER (3 nodes) │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ DNS-1 │ │ DNS-2 │ │ DNS-3 │ │
│ │ MASTER │ │ BACKUP │ │ BACKUP │ │
│ │ .54 │ │ .55 │ │ .56 │ │
│ └────┬────┘ └────┬────┘ └────┬────┘ │
│ └─────────────┼─────────────┘ │
│ ▼ │
│ VIP: 10.2.40.53 │
│ Failover: 3-5 seconds │
│ │
│ ───────────────────────────────────────────────────────────────────── │
│ │
│ POSTGRESQL HA (2 nodes + streaming replication) │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ authentik-db-1 │──────►│ authentik-db-2 │ │
│ │ PRIMARY │ repl │ STANDBY │ │
│ └────────┬────────┘ └────────┬────────┘ │
│ └────────────┬────────────┘ │
│ ▼ │
│ VIP: 10.2.40.135 │
│ Failover: automatic │
│ │
│ ───────────────────────────────────────────────────────────────────── │
│ │
│ RUSTFS S3 CLUSTER (3 nodes) │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ RFS-1 │ │ RFS-2 │ │ RFS-3 │ │
│ │ .120 │ │ .167 │ │ .218 │ │
│ └────┬────┘ └────┬────┘ └────┬────┘ │
│ └─────────────┼─────────────┘ │
│ ▼ │
│ VIP: s3.renner.uno │
│ Purpose: OpenTofu state backend │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Service Inventory
Core Infrastructure
| Service | Type | VMID | Node | Purpose |
|---|---|---|---|---|
| proxmox-dns-1 | LXC | 118 | jupiter | DNS resolution (MASTER) |
| proxmox-dns-2 | LXC | 218 | saturn | DNS resolution (BACKUP) |
| proxmox-dns-3 | LXC | 318 | neptune | DNS resolution (BACKUP) |
| rustfs-1/2/3 | LXC | 800+ | distributed | S3 state backend |
| cloudflared | LXC | 117 | - | Cloudflare tunnel |
Authentication & Identity
| Service | Type | VMID | Purpose |
|---|---|---|---|
| authentik-app-1 | LXC | 130 | Identity provider |
| authentik-app-2 | LXC | 131 | Identity provider (HA) |
| authentik-db-1 | LXC | 132 | PostgreSQL primary |
| authentik-db-2 | LXC | 133 | PostgreSQL standby |
Monitoring Stack
| Service | Type | VMID | Purpose |
|---|---|---|---|
| influxdb | VM | 108 | Time-series database |
| grafana | VM | 109 | Visualization (dash.renner.uno) |
| prometheus | LXC | - | Metrics collection |
| loki | LXC | - | Log aggregation |
| wazuh | LXC | 222 | Security monitoring |
Applications
| Service | Type | VMID | Purpose |
|---|---|---|---|
| gitea | LXC | 114 | Git server + CI/CD |
| immich | LXC | 225 | Photo management |
| paperless | LXC | - | Document management |
| home-assistant | VM | 103 | Home automation |
CI/CD
| Service | Type | VMID | Purpose |
|---|---|---|---|
| gitea-runner-1 | LXC | 905 | CI/CD execution |
| gitea-runner-2 | LXC | 906 | CI/CD execution |
Data Flow
DNS Resolution
Client Query: gitea.srv.renner.uno
│
▼
┌───────────────────┐
│ DNS VIP │
│ 10.2.40.53 │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Proxmox DNS │
│ (Custom Server) │
│ │
│ 1. Check cache │
│ 2. Query PVE API │
│ 3. Return IP │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Response: │
│ 10.2.40.x │
└───────────────────┘
Authentication Flow
User: https://grafana.renner.uno
│
▼
┌───────────────────┐
│ Caddy Ingress │
│ (TLS termination)│
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Grafana │
│ "Login with │
│ Authentik" │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Authentik │
│ (auth.renner.uno)│
│ │
│ - OIDC/OAuth2 │
│ - MFA (TOTP) │
│ - Group mapping │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Authenticated │
│ Session Created │
└───────────────────┘
Infrastructure Provisioning
Developer: git push
│
▼
┌───────────────────┐
│ Gitea │
│ (Trigger CI) │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Gitea Runner │
│ │
│ 1. tofu init │
│ 2. tofu plan │
│ 3. tofu apply │
└─────────┬─────────┘
│
▼
┌───────────────────┐ ┌───────────────────┐
│ RustFS S3 │◄─────►│ Proxmox API │
│ (State Backend) │ │ (Create VM/LXC) │
└───────────────────┘ └───────────────────┘
Dependency Map
┌─────────────┐
│ Internet │
└──────┬──────┘
│
┌──────▼──────┐
│ Cloudflare │
└──────┬──────┘
│
┌──────▼──────┐
│ Cloudflared │
└──────┬──────┘
│
┌───────────────────────────┼───────────────────────────┐
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Caddy Ingress │ │ DNS Cluster │◄────────│ Proxmox API │
│ (All traffic) │ │ (Name resolution) │ (Discovery) │
└────────┬────────┘ └────────┬────────┘ └─────────────────┘
│ │
│ │
▼ │
┌─────────────────┐ │
│ Authentik │◄─────────────────┘
│ (SSO/Identity)│
└────────┬────────┘
│
│ Protects
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ PROTECTED SERVICES │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Grafana │ │ Gitea │ │ Immich │ │Paperless │ │ ... │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Critical Dependencies
| Service | Depends On | Impact if Down |
|---|---|---|
| All services | DNS Cluster | No name resolution |
| All services | Caddy Ingress | No external access |
| Protected apps | Authentik | No authentication |
| OpenTofu | RustFS S3 | Cannot provision |
| Authentik | PostgreSQL HA | Auth database unavailable |
Backup Architecture
┌─────────────────────────────────────────────────────────────────────────┐
│ 3-TIER BACKUP STRATEGY │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ TIER 1: Proxmox VE Cluster (Source) │
│ ├── All VMs and LXC containers │
│ ├── Automated snapshot creation │
│ └── QEMU guest agent for consistency │
│ │ │
│ ▼ Automatic backup │
│ │
│ TIER 2: PBS2 (Primary Backup) │
│ ├── Deduplication + compression │
│ ├── Retention: 7 daily, 4 weekly, 12 monthly │
│ └── Separate NAS infrastructure │
│ │ │
│ ▼ Automatic replication │
│ │
│ TIER 3: pbs.obg (Offsite) │
│ ├── Geographic separation │
│ ├── Air-gapped from primary │
│ └── Ransomware protection │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Security Architecture
┌─────────────────────────────────────────────────────────────────────────┐
│ SECURITY LAYERS │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ LAYER 1: Network Perimeter │
│ ├── Cloudflare WAF + DDoS protection │
│ ├── UniFi firewall rules │
│ └── VLAN segmentation │
│ │
│ LAYER 2: Application Security │
│ ├── TLS everywhere (Let's Encrypt via Caddy) │
│ ├── Authentik SSO with MFA │
│ └── OAuth2/OIDC for service integration │
│ │
│ LAYER 3: Infrastructure Security │
│ ├── SSH key-based authentication only │
│ ├── Ansible Vault for secrets │
│ └── API tokens with minimal permissions │
│ │
│ LAYER 4: Monitoring & Detection │
│ ├── Wazuh SIEM │
│ ├── Trivy vulnerability scanning (weekly) │
│ └── Grafana alerting │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Technology Stack
Infrastructure as Code
| Tool | Purpose | State Storage |
|---|---|---|
| OpenTofu | Infrastructure provisioning | RustFS S3 |
| Ansible | Configuration management | Git (vault encrypted) |
| Gitea Actions | CI/CD automation | - |
Providers & Integrations
| Provider | Purpose |
|---|---|
| bpg/proxmox | VM/LXC management |
| ubiquiti-community/unifi | DNS records |
| hashicorp/aws | S3 backend access |
Runtime
| Component | Technology |
|---|---|
| Containers | Proxmox LXC (Debian) |
| VMs | Proxmox QEMU/KVM |
| Orchestration | Proxmox HA + keepalived |
| Storage | ZFS (local-zfs) |
Scaling Considerations
Current Capacity
- 3 Proxmox nodes
- 40+ services
- Adequate for homelab use
Growth Paths
-
Add Proxmox Node
- Join to existing cluster
- Automatic HA rebalancing
- Update OpenTofu node targets
-
Add Storage
- Expand ZFS pools
- Add Ceph for distributed storage
-
Geographic Distribution
- Site-to-site VPN
- Database replication across sites
- Multi-region RustFS
Related Documentation
CLAUDE.md- Project overview and commandsSECURITY.md- Access control and proceduresDISASTER_RECOVERY.md- Recovery runbooksBACKUP_INFRASTRUCTURE.md- Backup detailsansible/DOCUMENTATION.md- Ansible usage
Document maintained by infrastructure team. Update when architecture changes.