homelab/dns-server/HA_ARCHITECTURE.md

Proxmox DNS Server - High Availability Architecture

Overview

The DNS service is critical infrastructure and requires high availability. This design uses keepalived to provide a Virtual IP (VIP) with automatic failover.

Architecture

┌─────────────────────────────────────────────────────┐
│                  Clients                             │
│            (Query 10.2.40.53)                        │
└──────────────────┬──────────────────────────────────┘
                   │
                   ▼
        ┌──────────────────────┐
        │  Virtual IP (VIP)     │
        │    10.2.40.53         │
        │  (Managed by          │
        │   keepalived)         │
        └──────────┬─────────────┘
                   │
         ┌─────────┴─────────┐
         │                   │
         ▼                   ▼
┌─────────────────┐  ┌─────────────────┐
│  DNS Server 1   │  │  DNS Server 2   │
│  LXC 118        │  │  LXC 218        │
│  10.2.40.54     │  │  10.2.40.55     │
│  (MASTER)       │  │  (BACKUP)       │
│                 │  │                 │
│  Priority: 100  │  │  Priority: 90   │
└────────┬────────┘  └────────┬────────┘
         │                    │
         └──────────┬─────────┘
                    │
                    ▼
         ┌──────────────────────┐
         │   Proxmox Cluster    │
         │   (API Queries)      │
         └──────────────────────┘

IP Address Allocation

Resource	IP Address	Description
VIP	10.2.40.53	Virtual IP for DNS (client-facing)
DNS Server 1	10.2.40.54	Primary DNS server (LXC 118)
DNS Server 2	10.2.40.55	Secondary DNS server (LXC 218)
DNS Server 3*	10.2.40.56	Optional third instance (LXC 318)

*Third server is optional but recommended for production

Keepalived Configuration

VRRP Settings

• Virtual Router ID: 53 (matches DNS port)
• Authentication: Simple password (internal network)
• Check Interval: 2 seconds
• Priority:
  • Server 1 (Master): 100
  • Server 2 (Backup): 90
  • Server 3 (Backup): 80

Health Check

Keepalived monitors the DNS service using:

1. systemd status check: Is proxmox-dns.service running?
2. Port check: Is port 53 listening?
3. Query check: Can it resolve a test query?

If any check fails, the server reduces its priority, triggering failover.

Failover Behavior

Normal Operation

• Server 1 holds the VIP (10.2.40.53)
• Clients query VIP, reach Server 1
• Server 2 & 3 run but don't serve traffic

Server 1 Failure

1. Server 1 DNS service stops or LXC fails
2. Server 1 keepalived detects failure
3. Server 1 stops advertising VIP
4. Server 2 (higher priority than 3) claims VIP
5. Clients now reach Server 2 (transparent)
6. Failover time: ~3-5 seconds

Server 1 Recovery

1. Server 1 comes back online
2. Server 1 keepalived starts with priority 100
3. Server 1 reclaims VIP (preemptive)
4. Server 2 releases VIP
5. Clients return to Server 1

Split Brain Prevention

• VRRP multicast ensures servers coordinate
• Only one server holds VIP at a time
• If network partition occurs, both sides can serve DNS (degraded but working)

DNS Server Behavior

Each DNS server:

• Binds to 0.0.0.0:53 (all interfaces)
• Queries Proxmox API independently
• Maintains its own cache
• Returns identical results (stateless design)

Cache Consistency

Caches may differ slightly between servers (acceptable):

• Each server refreshes cache every 60s
• Cache inconsistency window: Max 60s
• DNS TTL is 300s, so clients cache anyway
• Impact: Negligible

Proxmox API Load

3 servers × 60s refresh = 3 queries/minute to Proxmox API

• Negligible load on Proxmox
• Each server makes 1 API call per minute

Deployment Topology

Option A: Distributed (Recommended)

• Server 1: jupiter
• Server 2: saturn
• Server 3: neptune

Pros: Survives node failure Cons: Requires network between nodes for VRRP

Option B: Collocated

• All servers on same node

Pros: Simple, fast VRRP Cons: Single point of failure (node failure = DNS down)

Recommendation: Use distributed topology for true HA.

Configuration Files

DNS Server (/etc/proxmox-dns/config.yaml)

dns_host: "0.0.0.0"  # Listen on all interfaces (VIP + real IP)
dns_port: 53
# ... rest of config same for all servers

Keepalived (/etc/keepalived/keepalived.conf)

vrrp_instance DNS_VIP {
    state MASTER  # or BACKUP
    interface eth0
    virtual_router_id 53
    priority 100  # 90 for backup, 80 for backup2

    virtual_ipaddress {
        10.2.40.53/24
    }

    track_script {
        check_dns
    }
}

vrrp_script check_dns {
    script "/usr/local/bin/check-dns-health.sh"
    interval 2
    weight -20  # Reduce priority by 20 if check fails
}

Monitoring

Check VIP Status

# Which server holds the VIP?
ip addr show | grep 10.2.40.53

# Keepalived status
systemctl status keepalived

# See failover logs
journalctl -u keepalived -f

Test Failover

# On current MASTER
systemctl stop proxmox-dns
# VIP should move to BACKUP within 3-5 seconds

# Verify from client
dig @10.2.40.53 test.renner.uno +short
# Should still work (now served by backup)

Health Check Script

#!/bin/bash
# /usr/local/bin/check-dns-health.sh

# Check systemd service
systemctl is-active proxmox-dns >/dev/null 2>&1 || exit 1

# Check port listening
ss -lun | grep -q ':53 ' || exit 1

# Try a test query
dig @localhost test.renner.uno +short +timeout=1 >/dev/null 2>&1
exit $?

Maintenance

Planned Maintenance (Graceful Failover)

# On MASTER server
# Step 1: Lower priority (triggers failover)
echo "MASTER -> BACKUP" > /etc/keepalived/priority
systemctl reload keepalived

# Step 2: Wait for VIP to move
ip addr show | grep 10.2.40.53  # Should be gone

# Step 3: Perform maintenance
systemctl stop proxmox-dns
apt update && apt upgrade
reboot

# Step 4: After maintenance, restore priority
echo "MASTER" > /etc/keepalived/priority
systemctl reload keepalived

Emergency: Remove Server from Pool

# Temporarily disable a server
systemctl stop keepalived

Scaling

To add more DNS servers:

1. Deploy new LXC with unique IP (e.g., 10.2.40.57)
2. Install proxmox-dns service
3. Install keepalived with lower priority
4. New server automatically joins HA pool

Recovery Scenarios

Scenario	Impact	Recovery
DNS service crash	3-5s outage	Automatic failover
LXC container crash	3-5s outage	Automatic failover
Network partition	Some clients may hit wrong server	Self-heals when network restored
Proxmox API down	All DNS fails (as expected)	Wait for Proxmox recovery
All DNS servers down	Total DNS failure	Manual intervention required

Cost

Resources per DNS server:

• RAM: 512MB
• CPU: 1 core
• Disk: 4GB

Total for 3-server HA:

• RAM: 1.5GB
• CPU: 3 cores (rarely used)
• Disk: 12GB

Very reasonable for critical infrastructure.