Eric Campbell 6 min read Infrastructure

HAProxy: High-Availability Load Balancing

HAProxy (High Availability Proxy) is the de facto standard for software load balancing in production environments. Unlike hardware load balancers that cost tens of thousands of dollars, HAProxy runs...

Key Insights

  • HAProxy excels at Layer 4 and Layer 7 load balancing with minimal resource overhead, handling hundreds of thousands of concurrent connections on commodity hardware while providing sub-millisecond routing decisions.
  • Health checks are your first line of defense against cascading failures—configure aggressive check intervals with proper rise/fall thresholds to automatically remove unhealthy backends before users notice degraded performance.
  • High availability requires more than just HAProxy configuration; implement VRRP-based failover with Keepalived and monitor both the HAProxy process and actual traffic flow to prevent split-brain scenarios.

Introduction to HAProxy and Load Balancing Fundamentals

HAProxy (High Availability Proxy) is the de facto standard for software load balancing in production environments. Unlike hardware load balancers that cost tens of thousands of dollars, HAProxy runs on standard Linux servers and delivers comparable—often superior—performance. It solves the fundamental problem of distributing traffic across multiple backend servers to prevent any single server from becoming a bottleneck or single point of failure.

Load balancing matters because modern applications demand both horizontal scalability and fault tolerance. When one backend server fails or becomes overwhelmed, HAProxy automatically routes traffic to healthy servers. When traffic spikes, you add more backends and HAProxy distributes the load without configuration changes.

The architecture is straightforward: frontends define how HAProxy listens for incoming connections (IP, port, protocol), backends specify pools of servers that handle requests, and health checks continuously verify backend availability. This separation allows flexible routing rules and centralized traffic management.

Basic HAProxy Configuration and Setup

Installation on Ubuntu/Debian is trivial:

sudo apt update
sudo apt install haproxy
sudo systemctl enable haproxy

HAProxy’s configuration lives in /etc/haproxy/haproxy.cfg. The file has four main sections: global for process-wide settings, defaults for shared configuration, frontend for listeners, and backend for server pools.

Here’s a minimal production-ready configuration for a web application:

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    maxconn 4000

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000ms
    timeout client  50000ms
    timeout server  50000ms

frontend web_frontend
    bind *:80
    default_backend web_servers
    option forwardfor
    http-request add-header X-Forwarded-Proto http

backend web_servers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    server web1 10.0.1.10:8080 check
    server web2 10.0.1.11:8080 check
    server web3 10.0.1.12:8080 check

This configuration listens on port 80 and distributes traffic across three backend servers. The option forwardfor directive preserves the client’s real IP address in the X-Forwarded-For header, critical for logging and security. Each backend server gets health-checked via HTTP GET requests to /health.

After editing the config, always validate before reloading:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy

Load Balancing Algorithms and Strategies

Choosing the right algorithm dramatically impacts performance and user experience. HAProxy offers several algorithms, each optimized for different scenarios:

roundrobin: Distributes requests sequentially. Simple and effective when backends have identical capacity. This is your default choice.

leastconn: Routes to the server with fewest active connections. Essential for applications with long-lived connections or variable request processing times (WebSockets, database queries, video streaming).

source: Hashes the client IP to consistently route the same client to the same backend. Useful for applications with server-side session state, though proper session management is better.

uri: Hashes the request URI to route similar requests to the same backend. Optimizes cache hit rates on backends.

Here’s how to configure different algorithms and session persistence:

backend api_servers
    balance leastconn
    option httpchk GET /api/health
    server api1 10.0.2.10:3000 check maxconn 100
    server api2 10.0.2.11:3000 check maxconn 100
    server api3 10.0.2.12:3000 check maxconn 100

backend websocket_servers
    balance leastconn
    option httpchk GET /ws/health
    timeout server 3600s
    server ws1 10.0.3.10:8080 check
    server ws2 10.0.3.11:8080 check

backend app_with_sessions
    balance roundrobin
    cookie SERVERID insert indirect nocache
    server app1 10.0.4.10:8080 check cookie app1
    server app2 10.0.4.11:8080 check cookie app2
    server app3 10.0.4.12:8080 check cookie app3

The cookie-based persistence inserts a SERVERID cookie that HAProxy uses to route subsequent requests from the same client to the same backend. The indirect option prevents the cookie from being sent to the backend server, and nocache ensures proxies don’t cache responses with the cookie.

Health Checks and Backend Server Management

Health checks prevent HAProxy from sending traffic to failed or degraded backends. Configure them aggressively—it’s better to temporarily remove a flaky server than route user requests to it.

HAProxy supports both Layer 4 (TCP) and Layer 7 (HTTP) health checks. HTTP checks are superior because they verify application health, not just network connectivity:

backend robust_health_checks
    balance roundrobin
    
    # HTTP health check with custom headers
    option httpchk GET /health HTTP/1.1\r\nHost:\ api.example.com
    http-check expect status 200
    http-check expect string "healthy"
    
    # Check every 2 seconds, consider down after 2 failures, up after 3 successes
    default-server inter 2s fall 2 rise 3
    
    # Backend servers with different health check settings
    server web1 10.0.1.10:8080 check
    server web2 10.0.1.11:8080 check
    server web3 10.0.1.12:8080 check backup
    
    # Maintenance mode - manually disable
    server web4 10.0.1.13:8080 check disabled

The inter parameter sets check intervals, fall defines consecutive failures before marking down, and rise specifies successful checks needed before marking up. The backup server only receives traffic when all primary servers are down—perfect for graceful degradation.

For critical applications, implement custom health check endpoints that verify dependencies:

# Flask health check endpoint example
@app.route('/health')
def health_check():
    try:
        # Verify database connection
        db.session.execute('SELECT 1')
        
        # Check Redis cache
        redis_client.ping()
        
        # Verify external API dependency
        if not external_api.is_reachable():
            return jsonify({'status': 'degraded'}), 503
            
        return jsonify({'status': 'healthy'}), 200
    except Exception as e:
        return jsonify({'status': 'unhealthy', 'error': str(e)}), 503

Achieving High Availability with HAProxy

A single HAProxy instance is still a single point of failure. High availability requires multiple HAProxy instances with automatic failover using Keepalived and VRRP (Virtual Router Redundancy Protocol).

The architecture uses a Virtual IP (VIP) that floats between HAProxy instances. Clients connect to the VIP, and Keepalived ensures only one instance owns it at any time. When the primary fails, the backup assumes the VIP within seconds.

Install Keepalived on both HAProxy servers:

sudo apt install keepalived

Configure the primary HAProxy server (/etc/keepalived/keepalived.conf):

vrrp_script check_haproxy {
    script "/usr/bin/killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    advert_int 1
    
    authentication {
        auth_type PASS
        auth_pass your_secure_password
    }
    
    virtual_ipaddress {
        192.168.1.100/24
    }
    
    track_script {
        check_haproxy
    }
}

Configure the backup server identically except:

    state BACKUP
    priority 100

The priority determines which server becomes master (higher wins). The vrrp_script monitors the HAProxy process and triggers failover if it dies.

For production, implement a more sophisticated health check:

#!/bin/bash
# /usr/local/bin/check_haproxy_health.sh

# Check if HAProxy process is running
if ! pgrep haproxy > /dev/null; then
    exit 1
fi

# Verify HAProxy is actually routing traffic
if ! curl -sf http://localhost:80/health > /dev/null; then
    exit 1
fi

exit 0

Update the Keepalived config:

vrrp_script check_haproxy {
    script "/usr/local/bin/check_haproxy_health.sh"
    interval 2
    fall 2
    rise 2
}

Monitoring, Logging, and Statistics

HAProxy’s built-in statistics page provides real-time visibility into traffic patterns, backend health, and performance metrics. Enable it with:

listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 5s
    stats auth admin:your_secure_password
    stats admin if TRUE

Access it at http://your-haproxy:8404/stats. The admin interface allows you to manually disable backends for maintenance without configuration changes.

For production monitoring, export metrics to Prometheus:

frontend prometheus
    bind *:8405
    http-request use-service prometheus-exporter if { path /metrics }

Configure comprehensive logging:

defaults
    log     global
    mode    http
    option  httplog
    option  log-separate-errors
    option  log-health-checks
    
    # Custom log format with timing information
    log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

Analyze logs for slow requests:

# Find requests taking over 1 second
awk '$9 > 1000' /var/log/haproxy.log

# Top 10 slowest endpoints
awk '{print $9, $NF}' /var/log/haproxy.log | sort -rn | head -10

Production Best Practices and Security Hardening

SSL/TLS termination at HAProxy offloads encryption from backends and centralizes certificate management:

frontend https_frontend
    bind *:443 ssl crt /etc/haproxy/certs/example.com.pem
    
    # Force HTTPS
    bind *:80
    redirect scheme https code 301 if !{ ssl_fc }
    
    # Security headers
    http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
    http-response set-header X-Frame-Options "SAMEORIGIN"
    http-response set-header X-Content-Type-Options "nosniff"
    
    # Rate limiting
    stick-table type ip size 100k expire 30s store http_req_rate(10s)
    http-request track-sc0 src
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 }
    
    # DDoS protection - connection limits
    acl too_many_connections src_conn_cur ge 10
    http-request deny deny_status 429 if too_many_connections
    
    default_backend web_servers

backend web_servers
    balance roundrobin
    option httpchk GET /health
    
    # Prevent backend overload
    server web1 10.0.1.10:8080 check maxconn 500
    server web2 10.0.1.11:8080 check maxconn 500
    server web3 10.0.1.12:8080 check maxconn 500

The certificate file should combine the certificate, intermediate certificates, and private key:

cat example.com.crt intermediate.crt example.com.key > /etc/haproxy/certs/example.com.pem
chmod 600 /etc/haproxy/certs/example.com.pem

Tune for high performance:

global
    maxconn 50000
    nbproc 1
    nbthread 4
    cpu-map auto:1/1-4 0-3
    tune.ssl.default-dh-param 2048
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

Use nbthread instead of nbproc for modern HAProxy versions—threads share memory more efficiently than processes. The cpu-map directive pins threads to specific CPU cores for better cache locality.

HAProxy is production-ready out of the box, but these configurations transform it into an enterprise-grade load balancer. Monitor your metrics, tune health checks for your specific application behavior, and test failover scenarios regularly. Your infrastructure’s reliability depends on it.

Liked this? There's more.

Every week: one practical technique, explained simply, with code you can use immediately.