Apache Spark - Coalesce vs Repartition Performance

Key Insights

• Coalesce avoids shuffle by combining partitions locally, making it significantly faster for reducing partition counts—but it can create severe data skew that hurts downstream performance
• Repartition triggers a full shuffle that evenly distributes data across partitions, costing more upfront but preventing skewed task execution times
• The right choice depends on what comes next: use coalesce before writing files, use repartition before joins or when partition balance matters for parallel processing

Introduction

Partition management is one of the most overlooked performance levers in Apache Spark. Your partition count directly determines parallelism—too few partitions and you underutilize cluster resources; too many and you drown in scheduling overhead and small file problems.

Spark provides two methods for adjusting partition counts: coalesce() and repartition(). They seem interchangeable at first glance, but they work fundamentally differently under the hood. Choosing wrong can mean the difference between a job that finishes in minutes and one that times out after hours.

This article breaks down exactly how each method works, when to use them, and the performance implications you need to understand.

How Repartition Works

repartition() performs a full shuffle of your data across the cluster. Every record gets assigned to a new partition based on either a hash of the entire row or a hash of specific columns you provide.

# Repartition to 100 partitions with random distribution
df_repartitioned = df.repartition(100)

# Repartition by specific column - records with same key go to same partition
df_repartitioned_by_key = df.repartition(100, col("customer_id"))

When you call explain() on a repartitioned DataFrame, you’ll see the shuffle stage clearly:

df.repartition(100).explain()

== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- Exchange RoundRobinPartitioning(100), REPARTITION_BY_NUM, [plan_id=123]
   +- Scan parquet [columns...]

That Exchange operator is the shuffle. Spark writes all data to disk, then reads it back into new partitions. Every executor sends data to every other executor over the network.

The cost is substantial: network I/O, disk I/O, serialization overhead, and memory pressure from shuffle buffers. For a 100GB dataset, you might see 100GB of shuffle write and 100GB of shuffle read in Spark UI.

The benefit is guaranteed even distribution. Each partition gets approximately total_records / num_partitions rows, regardless of how skewed your original data was.

How Coalesce Works

coalesce() takes a completely different approach. Instead of shuffling data across the cluster, it combines partitions that already exist on the same executor.

# Reduce from 1000 partitions to 10 without shuffle
df_coalesced = df.coalesce(10)

The explain() output shows no exchange operator:

df.coalesce(10).explain()

== Physical Plan ==
Coalesce 10
+- Scan parquet [columns...]

Coalesce creates what Spark calls a “narrow dependency”—each new partition depends on a subset of parent partitions, and no data moves between executors. Spark simply groups existing partitions together logically.

This makes coalesce dramatically faster than repartition when reducing partition counts. No network transfer, no disk spill, no serialization overhead.

The limitation: coalesce can only reduce partitions, never increase them. If you call coalesce(1000) on a DataFrame with 100 partitions, you still get 100 partitions.

Performance Benchmarks

Let’s measure the actual difference. Here’s a benchmark comparing both methods on the same dataset:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, rand
import time

spark = SparkSession.builder \
    .appName("CoalesceVsRepartition") \
    .config("spark.sql.shuffle.partitions", "200") \
    .getOrCreate()

# Create test dataset with 1000 partitions, ~10GB
df = spark.range(0, 100_000_000, numPartitions=1000) \
    .withColumn("value", rand()) \
    .withColumn("category", (col("id") % 100).cast("string"))

df.cache()
df.count()  # Materialize cache

# Benchmark coalesce
start = time.time()
df.coalesce(10).write.mode("overwrite").parquet("/tmp/coalesce_output")
coalesce_time = time.time() - start

# Benchmark repartition
start = time.time()
df.repartition(10).write.mode("overwrite").parquet("/tmp/repartition_output")
repartition_time = time.time() - start

print(f"Coalesce: {coalesce_time:.2f}s")
print(f"Repartition: {repartition_time:.2f}s")

Typical results on a 10-node cluster:

Repartition takes 4x longer and consumes significantly more resources. The shuffle dominates execution time.

However, these numbers only tell part of the story. What happens to downstream operations matters just as much.

Data Skew Considerations

Coalesce’s speed comes with a hidden cost: it doesn’t rebalance data. If your original partitions have uneven sizes, coalesce amplifies that imbalance.

# Check partition sizes after coalesce vs repartition
def get_partition_sizes(df):
    return df.rdd.glom().map(len).collect()

# Original data with skew (simulated)
skewed_df = spark.range(0, 10_000_000, numPartitions=100) \
    .withColumn("key", (col("id") % 10))  # 10 distinct keys

# Filter creates skew - most partitions now nearly empty
filtered_df = skewed_df.filter(col("key") == 0)

# Compare partition distributions
coalesce_sizes = get_partition_sizes(filtered_df.coalesce(10))
repartition_sizes = get_partition_sizes(filtered_df.repartition(10))

print(f"Coalesce partition sizes: {coalesce_sizes}")
print(f"Repartition partition sizes: {repartition_sizes}")

Output:

Coalesce partition sizes: [892451, 12, 8, 107529, 0, 0, 0, 0, 0, 0]
Repartition partition sizes: [100012, 99987, 100003, 99998, 100001, 99999, 100000, 100000, 100000, 100000]

Coalesce preserves the original data locality, which means some partitions get all the data while others sit empty. Repartition redistributes evenly.

This skew destroys parallelism. If you run a subsequent operation—say, a complex aggregation—on the coalesced DataFrame, one task processes 892,451 records while others process single digits. Your job runs as fast as the slowest task.

Check Spark UI’s task duration metrics. If you see one task taking 10 minutes while others finish in seconds, you have a skew problem that coalesce likely caused.

Decision Framework

Here’s when to use each method:

Use coalesce when:

• Reducing partitions before writing output files (you want fewer, larger files)
• Data is already reasonably balanced across partitions
• The operation immediately follows and no further transformations occur
• You’re reducing by a small factor (1000 → 100, not 1000 → 2)

Use repartition when:

• Increasing partition count (coalesce literally cannot do this)
• Preparing for a join on a specific key
• Data has become skewed from filters or aggregations
• Downstream operations require balanced parallel execution
• You need deterministic partitioning by column values

Here’s a real pipeline demonstrating appropriate usage:

# Raw data ingestion - 10,000 small files = 10,000 partitions
raw_df = spark.read.parquet("s3://bucket/raw_events/")

# Repartition by user_id before join - ensures co-location
events_by_user = raw_df.repartition(200, col("user_id"))

# Join with user dimension table (also partitioned by user_id)
users_df = spark.read.parquet("s3://bucket/users/") \
    .repartition(200, col("user_id"))

joined_df = events_by_user.join(users_df, "user_id")

# Heavy aggregation - benefits from balanced partitions
aggregated_df = joined_df.groupBy("user_id", "event_type") \
    .agg(
        count("*").alias("event_count"),
        sum("revenue").alias("total_revenue")
    )

# Coalesce before write - we want ~100MB files, not 200 tiny ones
# Data is already balanced from the aggregation
aggregated_df.coalesce(20) \
    .write \
    .mode("overwrite") \
    .parquet("s3://bucket/aggregated_events/")

The pattern: repartition early when you need balanced parallelism or key-based co-location, coalesce late when you’re just consolidating output files.

Conclusion

The coalesce vs repartition decision comes down to one trade-off: shuffle cost versus partition balance.

Coalesce is faster because it avoids shuffle, but it preserves whatever data distribution you already have—including skew. Repartition costs more upfront but guarantees even distribution that pays off in downstream operations.

Quick Reference:

Don’t guess—measure. Check Spark UI for shuffle metrics and task duration variance. If coalesce saves you 2 minutes on the coalesce operation but adds 10 minutes of skewed task execution downstream, you made the wrong choice.

The fastest Spark job isn’t the one that avoids shuffles at all costs. It’s the one that shuffles strategically to maintain balanced parallelism throughout the pipeline.