Eric Campbell 6 min read Engineering

Chaos Engineering: Resilience Testing

In 2011, Netflix engineers faced a problem: their systems had grown so complex that no one could confidently predict how they'd behave when things went wrong. Their solution was Chaos Monkey, a tool...

Key Insights

  • Chaos engineering is not about breaking things randomly—it’s a disciplined practice of forming hypotheses about system behavior and validating them through controlled failure injection
  • Start small with read-only experiments in staging environments, then gradually expand blast radius as you build confidence and organizational trust
  • The goal isn’t to prevent all failures but to reduce mean time to recovery (MTTR) by ensuring your team has practiced responding to realistic failure scenarios

Introduction to Chaos Engineering

In 2011, Netflix engineers faced a problem: their systems had grown so complex that no one could confidently predict how they’d behave when things went wrong. Their solution was Chaos Monkey, a tool that randomly terminated production instances during business hours. The reasoning was counterintuitive but sound—if you’re going to experience failures anyway, you might as well experience them on your terms.

Chaos engineering has since evolved from Netflix’s internal experiment into a formal discipline. The core principle remains unchanged: proactively inject failures into your systems to discover weaknesses before they become incidents. This isn’t about causing chaos—it’s about building confidence through controlled experimentation.

Most teams discover their systems are far more fragile than they assumed. Timeouts aren’t configured correctly. Circuit breakers never actually trip. Retry logic creates cascading failures. These issues hide in production until the worst possible moment. Chaos engineering surfaces them deliberately.

The Principles of Chaos Engineering

Chaos engineering borrows from the scientific method. You don’t just randomly kill services and see what happens. You form hypotheses and test them systematically.

The process follows four steps:

  1. Define steady state: Identify metrics that indicate normal system behavior. This might be request latency, error rates, throughput, or business metrics like orders per minute.

  2. Form a hypothesis: State what you believe will happen when you introduce a failure. “When database replica fails, the system will failover to the primary within 5 seconds with no user-visible errors.”

  3. Introduce variables: Inject the failure in a controlled manner. Start with the smallest blast radius that can validate your hypothesis.

  4. Analyze results: Compare actual behavior against your hypothesis. Did the system behave as expected? If not, you’ve found a weakness to address.

Blast radius is critical. You’re not trying to take down production—you’re trying to learn. Start with a single instance, a small percentage of traffic, or a non-critical service. Expand only after you’ve validated your safety mechanisms work.

Common Failure Injection Patterns

Real-world failures fall into predictable categories. Your chaos experiments should cover each:

Network failures include latency injection, packet loss, and network partitions. These are the most common production issues and often the least tested.

Service termination simulates crashed processes, killed containers, or failed instances. This validates your orchestration and load balancing.

Resource exhaustion covers CPU saturation, memory pressure, and disk space depletion. These slow-burn failures often cause the most confusing incidents.

Dependency failures test behavior when external services (databases, caches, third-party APIs) become unavailable or slow.

Here’s a practical example using Linux traffic control to inject network latency:

#!/bin/bash
# inject-latency.sh - Add 200ms latency to traffic on port 5432 (PostgreSQL)

INTERFACE="eth0"
DELAY="200ms"
JITTER="50ms"
TARGET_PORT="5432"
DURATION="60"

echo "Injecting ${DELAY} latency (±${JITTER}) to port ${TARGET_PORT} for ${DURATION}s"

# Add latency using tc (traffic control)
sudo tc qdisc add dev $INTERFACE root handle 1: prio
sudo tc qdisc add dev $INTERFACE parent 1:3 handle 30: netem delay $DELAY $JITTER
sudo tc filter add dev $INTERFACE protocol ip parent 1:0 prio 3 u32 \
    match ip dport $TARGET_PORT 0xffff flowid 1:3

# Wait for duration
sleep $DURATION

# Clean up
echo "Removing latency injection"
sudo tc qdisc del dev $INTERFACE root

echo "Experiment complete"

This script adds 200ms of latency (with 50ms of jitter) to all PostgreSQL traffic. Run it while monitoring your application’s response times and error rates to see how your system handles database slowdowns.

Building Your First Chaos Experiment

Let’s walk through a complete experiment targeting Kubernetes pod resilience. We’ll use Python with the Kubernetes client to terminate random pods and observe system behavior.

First, define your experiment parameters:

# chaos_experiment.py
import random
import time
from datetime import datetime
from kubernetes import client, config

class ChaosExperiment:
    def __init__(self, namespace: str, label_selector: str):
        config.load_kube_config()
        self.v1 = client.CoreV1Api()
        self.namespace = namespace
        self.label_selector = label_selector
        self.steady_state_metrics = {}
    
    def capture_steady_state(self) -> dict:
        """Capture metrics that define normal system behavior"""
        pods = self.v1.list_namespaced_pod(
            namespace=self.namespace,
            label_selector=self.label_selector
        )
        
        ready_pods = sum(
            1 for pod in pods.items 
            if pod.status.phase == "Running"
        )
        
        return {
            "timestamp": datetime.now().isoformat(),
            "total_pods": len(pods.items),
            "ready_pods": ready_pods,
            "ready_percentage": ready_pods / len(pods.items) * 100
        }
    
    def terminate_random_pod(self) -> str:
        """Kill a random pod matching the selector"""
        pods = self.v1.list_namespaced_pod(
            namespace=self.namespace,
            label_selector=self.label_selector
        )
        
        running_pods = [
            pod for pod in pods.items 
            if pod.status.phase == "Running"
        ]
        
        if not running_pods:
            raise Exception("No running pods found")
        
        victim = random.choice(running_pods)
        
        print(f"Terminating pod: {victim.metadata.name}")
        self.v1.delete_namespaced_pod(
            name=victim.metadata.name,
            namespace=self.namespace
        )
        
        return victim.metadata.name
    
    def wait_for_recovery(self, timeout_seconds: int = 120) -> bool:
        """Wait for system to return to steady state"""
        start_time = time.time()
        initial_count = self.steady_state_metrics["ready_pods"]
        
        while time.time() - start_time < timeout_seconds:
            current = self.capture_steady_state()
            
            if current["ready_pods"] >= initial_count:
                recovery_time = time.time() - start_time
                print(f"System recovered in {recovery_time:.2f} seconds")
                return True
            
            time.sleep(5)
        
        print(f"System did not recover within {timeout_seconds} seconds")
        return False
    
    def run_experiment(self):
        """Execute the full chaos experiment"""
        print("=== Chaos Experiment: Pod Termination ===")
        
        # Step 1: Capture steady state
        self.steady_state_metrics = self.capture_steady_state()
        print(f"Steady state: {self.steady_state_metrics}")
        
        # Step 2: Inject failure
        terminated_pod = self.terminate_random_pod()
        
        # Step 3: Observe and measure recovery
        recovered = self.wait_for_recovery(timeout_seconds=120)
        
        # Step 4: Report results
        final_state = self.capture_steady_state()
        print(f"Final state: {final_state}")
        print(f"Experiment {'PASSED' if recovered else 'FAILED'}")
        
        return recovered


if __name__ == "__main__":
    experiment = ChaosExperiment(
        namespace="production",
        label_selector="app=api-gateway"
    )
    experiment.run_experiment()

Before running this, establish your hypothesis: “When a single API gateway pod is terminated, Kubernetes will schedule a replacement and the system will return to full capacity within 60 seconds with no failed requests.”

Chaos Engineering Tools Landscape

While custom scripts work for simple experiments, dedicated tools provide better safety controls, observability, and experiment management.

Chaos Mesh is a CNCF project that integrates deeply with Kubernetes. Here’s a configuration that injects network latency into a specific service:

# network-delay-experiment.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: api-gateway-latency
  namespace: chaos-testing
spec:
  action: delay
  mode: all
  selector:
    namespaces:
      - production
    labelSelectors:
      app: api-gateway
  delay:
    latency: "100ms"
    correlation: "25"
    jitter: "25ms"
  duration: "5m"
  scheduler:
    cron: "@every 24h"

LitmusChaos offers a broader experiment library and works well in multi-cloud environments. Gremlin provides a commercial solution with excellent safety controls and a web UI—worth considering if you need enterprise features. AWS Fault Injection Simulator integrates natively with AWS services if you’re fully committed to that ecosystem.

Choose based on your infrastructure. Kubernetes-native teams should evaluate Chaos Mesh or LitmusChaos. Multi-cloud or hybrid environments benefit from Gremlin’s flexibility.

Integrating Chaos into CI/CD

Chaos experiments become most valuable when they run automatically. Here’s a GitHub Actions workflow that runs chaos experiments after deployment:

# .github/workflows/chaos-testing.yml
name: Post-Deployment Chaos Testing

on:
  workflow_run:
    workflows: ["Deploy to Staging"]
    types: [completed]

jobs:
  chaos-experiments:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Configure kubectl
        uses: azure/k8s-set-context@v3
        with:
          kubeconfig: ${{ secrets.KUBE_CONFIG_STAGING }}
      
      - name: Install Chaos Mesh
        run: |
          helm repo add chaos-mesh https://charts.chaos-mesh.org
          helm upgrade --install chaos-mesh chaos-mesh/chaos-mesh \
            --namespace chaos-testing --create-namespace          
      
      - name: Run Pod Failure Experiment
        run: |
          kubectl apply -f chaos-experiments/pod-failure.yaml
          sleep 300  # Run for 5 minutes
                    
      - name: Verify System Health
        run: |
          # Check that error rate stayed below threshold
          ERROR_RATE=$(curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{status=~'5..'}[5m])" | jq '.data.result[0].value[1]')
          if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
            echo "Error rate exceeded 1% during chaos experiment"
            exit 1
          fi          
      
      - name: Cleanup Experiments
        if: always()
        run: kubectl delete -f chaos-experiments/ --ignore-not-found

Start with staging environments. Only graduate to production chaos after you’ve validated your experiments are safe and your team is comfortable with the process. Many organizations run “gamedays”—scheduled chaos sessions where the team actively monitors and responds to injected failures.

Measuring Success and Organizational Adoption

Track these metrics to demonstrate chaos engineering value:

  • Mean Time to Recovery (MTTR): Should decrease as teams practice incident response
  • Incident frequency: Fewer surprises in production as weaknesses are found proactively
  • Time to detect: Monitoring gaps surface during experiments

Getting organizational buy-in requires addressing legitimate concerns. “You want to break production on purpose?” is a reasonable objection. Counter it with data: unplanned outages cost more than controlled experiments. Frame chaos engineering as practice, not recklessness.

Start with read-only experiments—observing system behavior under load without injecting failures. Graduate to failure injection in staging. Only move to production after demonstrating safety controls work and the team has built confidence.

Document every experiment. Share results widely. When chaos experiments catch issues before they become incidents, celebrate those wins publicly. Nothing builds organizational support faster than preventing a 3 AM page.

Chaos engineering isn’t about proving your systems are fragile—it’s about making them stronger through deliberate practice. Start small, measure everything, and expand gradually. Your future self, paged at 2 AM for an incident you’ve already practiced handling, will thank you.

Liked this? There's more.

Every week: one practical technique, explained simply, with code you can use immediately.