Eric Campbell 6 min read Infrastructure

Auto-Scaling: Horizontal and Vertical Scaling Strategies

Auto-scaling automatically adjusts computational resources based on actual demand, preventing both resource waste during low traffic and performance degradation during spikes. Without auto-scaling,...

Key Insights

  • Horizontal scaling provides better fault tolerance and theoretically unlimited capacity but requires stateless architecture and load balancing, while vertical scaling is simpler to implement but hits hard limits and creates single points of failure.
  • Most production systems benefit from a hybrid approach: vertical scaling for stateful components like databases, horizontal scaling for stateless application layers, with metrics-driven automation to handle traffic variations.
  • Proper scaling implementation depends on choosing the right metrics (CPU, memory, request latency, or custom business metrics), setting appropriate thresholds with cooldown periods, and designing applications to be stateless from day one.

Introduction to Auto-Scaling

Auto-scaling automatically adjusts computational resources based on actual demand, preventing both resource waste during low traffic and performance degradation during spikes. Without auto-scaling, you’re either over-provisioning and burning money, or under-provisioning and losing customers during peak loads.

There are two fundamental approaches: vertical scaling (adding more power to existing machines) and horizontal scaling (adding more machines). Vertical scaling means upgrading from a 4-core to an 8-core instance. Horizontal scaling means running three 4-core instances instead of one. Each has distinct trade-offs that make them suitable for different scenarios.

Choose vertical scaling when you have stateful applications, legacy systems that can’t distribute across nodes, or when you need the simplest possible solution. Choose horizontal scaling for stateless web applications, microservices, or when you need high availability and fault tolerance. In practice, most architectures use both.

Vertical Scaling (Scale Up/Down)

Vertical scaling adds CPU, RAM, or disk resources to existing instances. It’s conceptually simple: when your database server runs out of memory, you upgrade to a larger instance type. No code changes, no distributed system complexity.

The primary advantage is simplicity. Your application doesn’t need to know about multiple instances, you don’t need load balancers, and session management remains straightforward. For databases and stateful applications, vertical scaling is often the only practical option without significant architectural changes.

However, vertical scaling has hard limits. You can’t infinitely increase a single machine’s resources—eventually you hit the largest instance type your cloud provider offers. Scaling operations typically require downtime or at least a brief service interruption. You also create a single point of failure: if that one big instance fails, your entire service goes down.

Here’s how to modify an EC2 instance type with AWS CLI:

# Stop the instance
aws ec2 stop-instances --instance-ids i-1234567890abcdef0

# Wait for stopped state
aws ec2 wait instance-stopped --instance-ids i-1234567890abcdef0

# Modify instance type
aws ec2 modify-instance-attribute \
  --instance-id i-1234567890abcdef0 \
  --instance-type "{\"Value\": \"m5.2xlarge\"}"

# Start the instance
aws ec2 start-instances --instance-ids i-1234567890abcdef0

For Kubernetes workloads, the Vertical Pod Autoscaler adjusts resource requests automatically:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: api-server
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: api
      minAllowed:
        cpu: 100m
        memory: 128Mi
      maxAllowed:
        cpu: 2
        memory: 4Gi

Horizontal Scaling (Scale Out/In)

Horizontal scaling adds or removes identical instances of your application. When traffic increases, you spin up more containers or VMs. When it decreases, you terminate excess instances. This approach provides true elasticity and fault tolerance.

The advantages are compelling: no hard ceiling on capacity, built-in redundancy, and zero-downtime deployments. If one instance fails, others continue serving traffic. You can scale to thousands of instances if needed.

The requirements are stricter: your application must be stateless, or at least store state externally in databases or caches. You need a load balancer to distribute traffic. Session data must be shared across instances using Redis, database-backed sessions, or JWT tokens.

Here’s a Kubernetes Horizontal Pod Autoscaler configuration:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 30

For AWS, here’s a Terraform configuration for an Auto Scaling Group:

resource "aws_autoscaling_group" "api" {
  name                = "api-asg"
  vpc_zone_identifier = var.private_subnet_ids
  target_group_arns   = [aws_lb_target_group.api.arn]
  health_check_type   = "ELB"
  health_check_grace_period = 300

  min_size         = 2
  max_size         = 10
  desired_capacity = 2

  launch_template {
    id      = aws_launch_template.api.id
    version = "$Latest"
  }

  tag {
    key                 = "Name"
    value               = "api-server"
    propagate_at_launch = true
  }
}

resource "aws_autoscaling_policy" "scale_up" {
  name                   = "api-scale-up"
  scaling_adjustment     = 2
  adjustment_type        = "ChangeInCapacity"
  cooldown              = 300
  autoscaling_group_name = aws_autoscaling_group.api.name
}

resource "aws_autoscaling_policy" "scale_down" {
  name                   = "api-scale-down"
  scaling_adjustment     = -1
  adjustment_type        = "ChangeInCapacity"
  cooldown              = 300
  autoscaling_group_name = aws_autoscaling_group.api.name
}

Implementing Scaling Metrics and Triggers

Choosing the right metrics determines whether your auto-scaling actually works. CPU utilization is the default, but it’s often not the best indicator of application health. A CPU-bound application might scale perfectly on CPU metrics, but an I/O-bound application won’t.

Better metrics include request latency (scale when response times degrade), request count (scale based on actual load), or custom business metrics (active users, queue depth, order processing rate). Memory utilization matters for memory-intensive applications. Custom metrics give you precise control.

Set thresholds conservatively. Scaling at 90% CPU utilization is too late—your users are already experiencing degraded performance. Scale at 60-70% to maintain headroom. Always configure cooldown periods to prevent rapid oscillation where instances constantly spin up and down.

Here’s how to publish custom CloudWatch metrics in Python:

import boto3
from datetime import datetime

cloudwatch = boto3.client('cloudwatch')

def publish_queue_depth(queue_name, depth):
    cloudwatch.put_metric_data(
        Namespace='CustomApp/Processing',
        MetricData=[
            {
                'MetricName': 'QueueDepth',
                'Dimensions': [
                    {
                        'Name': 'QueueName',
                        'Value': queue_name
                    },
                ],
                'Value': depth,
                'Unit': 'Count',
                'Timestamp': datetime.utcnow()
            },
        ]
    )

# Use in your application
current_depth = get_queue_depth('orders')
publish_queue_depth('orders', current_depth)

For Kubernetes with custom metrics, configure HPA to use Prometheus:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-custom-metrics
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"

Hybrid Scaling Strategies

Real-world architectures combine both scaling approaches. Your database vertically scales (larger RDS instances), your application layer horizontally scales (more containers), and your cache layer does both (Redis cluster with larger nodes).

This makes sense because different components have different constraints. Databases are inherently stateful and difficult to horizontally scale without sharding. Application servers should be stateless and scale horizontally. Caches can often do both—vertical scaling for hot data sets, horizontal scaling for distribution.

Cost optimization drives hybrid strategies. Horizontal scaling with many small instances provides fine-grained control and better resource utilization. Vertical scaling reduces networking overhead and simplifies architecture. The right balance depends on your traffic patterns and budget.

Here’s a Terraform example showing multi-tier scaling:

# Database: Vertical scaling via instance class
resource "aws_db_instance" "main" {
  identifier     = "app-db"
  instance_class = var.db_instance_class  # Manually adjusted
  engine         = "postgres"
  engine_version = "15.3"
  
  allocated_storage = 100
  storage_type      = "gp3"
  
  # Vertical scaling only
  apply_immediately = false
}

# Application: Horizontal scaling via ECS
resource "aws_ecs_service" "api" {
  name            = "api-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.api.arn
  desired_count   = 2
  
  load_balancer {
    target_group_arn = aws_lb_target_group.api.arn
    container_name   = "api"
    container_port   = 8080
  }
}

resource "aws_appautoscaling_target" "ecs_target" {
  max_capacity       = 10
  min_capacity       = 2
  resource_id        = "service/${aws_ecs_cluster.main.name}/${aws_ecs_service.api.name}"
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace  = "ecs"
}

resource "aws_appautoscaling_policy" "ecs_policy" {
  name               = "api-scaling-policy"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_appautoscaling_target.ecs_target.resource_id
  scalable_dimension = aws_appautoscaling_target.ecs_target.scalable_dimension
  service_namespace  = aws_appautoscaling_target.ecs_target.service_namespace

  target_tracking_scaling_policy_configuration {
    target_value       = 70.0
    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageCPUUtilization"
    }
    scale_in_cooldown  = 300
    scale_out_cooldown = 60
  }
}

Best Practices and Common Pitfalls

Design for statelessness from day one. Store sessions in Redis or use JWT tokens. Don’t rely on local disk or in-memory state. This single decision determines whether you can horizontally scale.

Database scaling remains the hardest problem. Read replicas help with read-heavy workloads. Connection pooling prevents overwhelming your database when application instances scale up. Consider whether you actually need strong consistency, or if eventual consistency allows you to use more scalable data stores.

Avoid scaling oscillation by setting appropriate cooldown periods and using gradual scaling policies. Scaling up aggressively but down conservatively prevents constant churn. The behavior policies in the Kubernetes HPA example above demonstrate this: scale up by 100% every 30 seconds if needed, but scale down by only 50% per minute with a 5-minute stabilization window.

Test your scaling policies before production. Simulate traffic spikes with load testing tools like k6 or Gatling. Verify that scaling triggers at expected thresholds and that new instances become healthy quickly. Monitor scaling events and tune your policies based on actual behavior.

Implement comprehensive monitoring. Track not just whether scaling occurs, but whether it improves performance. Monitor scaling lag (time from trigger to new capacity), instance health, and application-level metrics during scaling events.

Conclusion

Choose vertical scaling for simplicity and stateful components, horizontal scaling for fault tolerance and elastic capacity. Most production systems need both: vertically scale your databases and stateful services, horizontally scale your stateless application tier.

The future points toward serverless and edge computing, where scaling becomes even more granular and automatic. Functions scale to zero when unused and to thousands of concurrent executions automatically. Edge computing distributes workloads globally, scaling closer to users.

Key takeaways: Design applications to be stateless, choose metrics that reflect actual user experience, set conservative thresholds with appropriate cooldown periods, test scaling policies under realistic load, and monitor everything. Auto-scaling isn’t a silver bullet, but implemented correctly, it provides both cost efficiency and reliability.

Liked this? There's more.

Every week: one practical technique, explained simply, with code you can use immediately.