Eric Campbell 6 min read Engineering

Apache Spark - Executor Memory and Cores Configuration

Apache Spark's performance lives or dies by how you configure executor memory and cores. Get it wrong, and you'll watch jobs crawl through excessive garbage collection, crash with cryptic...

Key Insights

  • Executor memory is divided into execution, storage, user, and reserved regions—understanding this architecture prevents most out-of-memory errors and performance bottlenecks.
  • The “5 cores per executor” rule exists because HDFS clients struggle with more than 5 concurrent threads, but you should adjust based on your storage layer and workload characteristics.
  • Memory overhead (typically 10% of executor memory or 384MB minimum) is allocated outside the JVM heap for native libraries and container management—ignoring it causes silent container kills.

Introduction to Spark Resource Management

Apache Spark’s performance lives or dies by how you configure executor memory and cores. Get it wrong, and you’ll watch jobs crawl through excessive garbage collection, crash with cryptic out-of-memory errors, or waste cluster resources with underutilized executors.

Spark distributes work across executors—JVM processes that run on worker nodes. Each executor receives a portion of your cluster’s memory and CPU cores. The driver divides your job into tasks, and executors run these tasks in parallel using their allocated cores. One core handles one task at a time.

The challenge is that there’s no universal “correct” configuration. Optimal settings depend on your cluster size, workload type, data characteristics, and resource manager. This article gives you the mental model and practical formulas to make informed decisions rather than cargo-culting configurations from Stack Overflow.

Understanding Executor Memory Architecture

Spark’s unified memory model, introduced in version 1.6, divides executor memory into four regions:

Reserved Memory: Fixed at 300MB. Spark uses this for internal objects and prevents you from running executors smaller than 450MB.

User Memory: Stores your data structures, UDF variables, and RDD metadata. Calculated as (Executor Memory - 300MB) × (1 - spark.memory.fraction).

Unified Memory: The remaining space, split dynamically between execution and storage based on demand:

  • Execution Memory: Holds intermediate data during shuffles, joins, sorts, and aggregations.
  • Storage Memory: Caches RDDs, broadcast variables, and DataFrames.

By default, spark.memory.fraction is 0.6, meaning 60% of usable memory goes to the unified pool. The spark.memory.storageFraction parameter (default 0.5) sets the boundary between storage and execution within that pool—but this boundary is soft. Execution can evict cached data if it needs more space.

# Basic memory configuration via spark-submit
spark-submit \
  --executor-memory 8g \
  --conf spark.memory.fraction=0.6 \
  --conf spark.memory.storageFraction=0.5 \
  --class com.example.MyApp \
  myapp.jar

# For a 8GB executor, memory breakdown:
# Reserved: 300MB (fixed)
# Usable: 8192MB - 300MB = 7892MB
# Unified Memory: 7892MB × 0.6 = 4735MB
# User Memory: 7892MB × 0.4 = 3157MB

When execution memory fills up, Spark spills data to disk. When storage memory is exhausted, cached data gets evicted. Both scenarios hurt performance, but execution memory pressure is usually worse—it means your shuffles and aggregations are thrashing.

Configuring Executor Cores

Each executor core runs one task concurrently. More cores per executor means more parallel tasks sharing that executor’s memory. This creates a fundamental tradeoff.

More cores per executor (e.g., 8 cores, 32GB memory):

  • Better memory sharing for cached data
  • Reduced overhead from fewer JVM processes
  • Risk of GC pauses affecting all 8 concurrent tasks
  • HDFS throughput degradation beyond 5 concurrent threads

Fewer cores per executor (e.g., 2 cores, 8GB memory):

  • Isolated failure domains
  • More predictable memory usage per task
  • Higher aggregate overhead from more JVM processes
  • Better for memory-intensive tasks
# Executor core configuration
spark-submit \
  --executor-cores 5 \
  --conf spark.task.cpus=1 \
  --conf spark.default.parallelism=200 \
  --executor-memory 20g \
  --class com.example.MyApp \
  myapp.jar

# spark.task.cpus controls cores per task (default 1)
# Useful for multi-threaded libraries like XGBoost
spark-submit \
  --executor-cores 8 \
  --conf spark.task.cpus=4 \
  --executor-memory 32g \
  --class com.example.MLTraining \
  mlapp.jar
# This runs 2 tasks per executor (8 cores ÷ 4 cpus/task)

The relationship between cores, tasks, and partitions is straightforward: Spark creates one task per partition, and tasks run on available cores. If you have 100 partitions and 20 total executor cores across your cluster, you’ll run 20 tasks concurrently, processing all 100 partitions in 5 waves.

Memory Overhead and Off-Heap Considerations

Here’s where many configurations fail silently. The memory you request with --executor-memory is just JVM heap. Your executor also needs memory for:

  • JVM internals (metaspace, thread stacks, code cache)
  • Native libraries (compression codecs, serialization)
  • Container management overhead
  • Off-heap storage if enabled

YARN and Kubernetes allocate containers based on total memory, not just heap. If your container exceeds its limit, the resource manager kills it—often without a clear error message.

# YARN configuration with explicit overhead
spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --executor-memory 16g \
  --conf spark.executor.memoryOverhead=2g \
  --conf spark.executor.cores=5 \
  --class com.example.MyApp \
  myapp.jar

# Total container memory = 16GB + 2GB = 18GB
# YARN will allocate an 18GB container

# Kubernetes configuration
spark-submit \
  --master k8s://https://k8s-master:6443 \
  --deploy-mode cluster \
  --executor-memory 16g \
  --conf spark.kubernetes.executor.request.cores=5 \
  --conf spark.kubernetes.memoryOverheadFactor=0.1 \
  --conf spark.executor.memoryOverhead=2g \
  --class com.example.MyApp \
  myapp.jar

For off-heap memory (useful for avoiding GC overhead with large datasets):

# Enable off-heap memory
spark-submit \
  --executor-memory 12g \
  --conf spark.memory.offHeap.enabled=true \
  --conf spark.memory.offHeap.size=4g \
  --conf spark.executor.memoryOverhead=3g \
  --class com.example.MyApp \
  myapp.jar

# Total memory footprint: 12GB heap + 4GB off-heap + 3GB overhead = 19GB

The default overhead calculation is max(384MB, 0.1 × executor-memory). For a 16GB executor, that’s 1.6GB. Increase it if you use native libraries heavily or see containers getting killed without heap OOM errors.

Practical Sizing Strategies

Start with your cluster resources and work backward. Here’s a systematic approach:

def calculate_spark_config(
    total_nodes: int,
    cores_per_node: int,
    memory_per_node_gb: int,
    os_reserve_gb: int = 1,
    cores_for_os: int = 1
) -> dict:
    """
    Calculate recommended Spark executor configuration.
    
    Uses the 5-cores-per-executor heuristic with adjustments.
    """
    # Reserve resources for OS and node manager
    available_cores = cores_per_node - cores_for_os
    available_memory = memory_per_node_gb - os_reserve_gb
    
    # Target 5 cores per executor (HDFS optimization)
    cores_per_executor = min(5, available_cores)
    executors_per_node = available_cores // cores_per_executor
    
    # Calculate memory per executor
    memory_per_executor = available_memory // executors_per_node
    
    # Account for overhead (10% or 384MB minimum)
    overhead = max(0.384, memory_per_executor * 0.1)
    heap_memory = memory_per_executor - overhead
    
    # Total executors (reserve 1 for driver in cluster mode)
    total_executors = (executors_per_node * total_nodes) - 1
    
    return {
        "executor_cores": cores_per_executor,
        "executor_memory_gb": round(heap_memory, 1),
        "executor_overhead_gb": round(overhead, 1),
        "num_executors": total_executors,
        "total_cores": total_executors * cores_per_executor,
        "parallelism": total_executors * cores_per_executor * 2
    }

# Example: 10-node cluster, 16 cores and 64GB RAM each
config = calculate_spark_config(
    total_nodes=10,
    cores_per_node=16,
    memory_per_node_gb=64
)
print(config)
# Output:
# {
#   'executor_cores': 5,
#   'executor_memory_gb': 18.9,
#   'executor_overhead_gb': 2.1,
#   'num_executors': 29,
#   'total_cores': 145,
#   'parallelism': 290
# }

When to deviate from the 5-core rule:

  • Cloud object storage (S3, GCS): These handle concurrency better than HDFS. You can safely use 8-10 cores per executor.
  • Memory-intensive ML workloads: Use 2-3 cores with more memory per core.
  • Streaming applications: Smaller executors (2-3 cores) provide faster recovery from failures.

Diagnosing Memory and Core Issues

The Spark UI tells you almost everything you need to know. Key metrics to watch:

Executors Tab:

  • Storage Memory: Should show consistent usage if you’re caching
  • Shuffle Read/Write: High values indicate data movement bottlenecks
  • GC Time: Over 10% of task time suggests memory pressure

Stages Tab:

  • Task Duration distribution: Wide variance indicates data skew
  • Shuffle Spill (Memory/Disk): Any disk spill means execution memory is insufficient
# Enable detailed GC logging
spark-submit \
  --executor-memory 16g \
  --conf "spark.executor.extraJavaOptions=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/tmp/gc-executor.log" \
  --conf "spark.driver.extraJavaOptions=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" \
  --class com.example.MyApp \
  myapp.jar

# For Java 11+, use unified logging
spark-submit \
  --executor-memory 16g \
  --conf "spark.executor.extraJavaOptions=-Xlog:gc*:file=/tmp/gc-executor.log:time,uptime,level,tags" \
  --class com.example.MyApp \
  myapp.jar

Common symptoms and fixes:

  • OOM on executor: Increase --executor-memory or reduce spark.executor.cores
  • Container killed by YARN: Increase spark.executor.memoryOverhead
  • Excessive GC: Reduce cores per executor or increase memory
  • Low CPU utilization: Increase parallelism or add more cores per executor

Configuration Best Practices Summary

# spark-defaults.conf - ETL workloads
# Optimized for shuffle-heavy transformations

spark.executor.memory              20g
spark.executor.cores               5
spark.executor.memoryOverhead      2g
spark.driver.memory                4g
spark.driver.cores                 2

# Memory tuning
spark.memory.fraction              0.6
spark.memory.storageFraction       0.5

# Parallelism (2-3x total cores)
spark.default.parallelism          300
spark.sql.shuffle.partitions       300

# Shuffle optimization
spark.shuffle.compress             true
spark.shuffle.spill.compress       true

# Dynamic allocation (recommended for shared clusters)
spark.dynamicAllocation.enabled           true
spark.dynamicAllocation.minExecutors      2
spark.dynamicAllocation.maxExecutors      50
spark.dynamicAllocation.executorIdleTimeout  60s

For ML workloads, increase memory per core and consider enabling off-heap. For streaming, use smaller executors with faster recovery. Test your configuration with representative data volumes—what works for 10GB will fail at 10TB.

The configurations in this article are starting points. Profile your actual workloads, watch the Spark UI, and iterate. Resource tuning is empirical, not theoretical.

Liked this? There's more.

Every week: one practical technique, explained simply, with code you can use immediately.