Eric Campbell 6 min read Engineering

Apache Spark vs Apache Flink

The big data processing landscape has consolidated around two dominant frameworks: Apache Spark and Apache Flink. Both can handle batch and stream processing, both scale horizontally, and both have...

Key Insights

  • Apache Flink’s true streaming architecture delivers lower latency and superior state management for real-time applications, while Spark’s micro-batch model offers simpler operations and better batch processing maturity.
  • Choose Spark when your workload is primarily batch-oriented with occasional streaming, your team already knows the ecosystem, or you need tight integration with the Hadoop/Databricks stack.
  • Choose Flink when sub-second latency is non-negotiable, you need complex event processing with large state, or you’re building event-driven architectures where streaming is the primary paradigm.

Introduction

The big data processing landscape has consolidated around two dominant frameworks: Apache Spark and Apache Flink. Both can handle batch and stream processing, both scale horizontally, and both have active communities. Yet they approach the problem from fundamentally different directions, and that difference matters more than most comparison articles admit.

Spark emerged from UC Berkeley’s AMPLab in 2009, designed to overcome MapReduce’s limitations by keeping data in memory. It treats streaming as a special case of batch processing. Flink came from the Stratosphere research project at TU Berlin, built from the ground up as a streaming engine that happens to handle batch workloads. This philosophical difference permeates every design decision in both frameworks.

Choosing between them isn’t about which is “better”—it’s about which aligns with your actual workload patterns, latency requirements, and operational capabilities.

Architecture Comparison

Spark’s execution model centers on Resilient Distributed Datasets (RDDs) and the higher-level DataFrame/Dataset APIs. When processing streams, Spark Structured Streaming divides incoming data into micro-batches, processes each batch using the same engine that handles batch jobs, and outputs results. This micro-batch approach typically introduces latency measured in hundreds of milliseconds to seconds.

Flink processes records one at a time as they arrive. Its dataflow model treats data as continuous streams, with batch processing implemented as a bounded stream. This record-at-a-time processing enables true low-latency streaming with latencies in the low milliseconds.

Here’s how a basic job setup reveals these architectural differences:

# Apache Spark - Structured Streaming
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("SparkStreamingJob") \
    .config("spark.sql.streaming.checkpointLocation", "/checkpoint") \
    .getOrCreate()

# Read from Kafka - processes in micro-batches
df = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "events") \
    .load()

# Micro-batch trigger - minimum 100ms practical latency
query = df.writeStream \
    .trigger(processingTime="1 second") \
    .format("console") \
    .start()

// Apache Flink - DataStream API
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.connector.kafka.source.KafkaSource;

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

// True record-at-a-time processing
KafkaSource<String> source = KafkaSource.<String>builder()
    .setBootstrapServers("localhost:9092")
    .setTopics("events")
    .setValueOnlyDeserializer(new SimpleStringSchema())
    .build();

// Continuous processing - millisecond latency achievable
env.fromSource(source, WatermarkStrategy.noWatermarks(), "Kafka Source")
    .print();

env.execute("FlinkStreamingJob");

Memory management differs significantly too. Spark relies heavily on JVM garbage collection, though Project Tungsten introduced off-heap memory management for certain operations. Flink manages memory explicitly with its own memory segments, giving it more predictable performance under pressure and better control over state storage.

Stream Processing Capabilities

Stream processing is where Flink genuinely excels. Its event-time processing, watermark handling, and windowing semantics are more sophisticated than Spark’s equivalents.

Consider windowed aggregations—a common streaming pattern:

# Spark Structured Streaming - Tumbling Window
from pyspark.sql.functions import window, sum, col

windowed_counts = df \
    .withWatermark("event_time", "10 minutes") \
    .groupBy(
        window(col("event_time"), "5 minutes"),
        col("user_id")
    ) \
    .agg(sum("amount").alias("total_amount"))

# Sliding window
sliding_counts = df \
    .withWatermark("event_time", "10 minutes") \
    .groupBy(
        window(col("event_time"), "10 minutes", "5 minutes"),
        col("user_id")
    ) \
    .agg(sum("amount").alias("total_amount"))

// Flink DataStream API - Tumbling Window
import org.apache.flink.streaming.api.windowing.assigners.*;
import org.apache.flink.streaming.api.windowing.time.Time;

DataStream<Event> events = // ... source

// Tumbling window with event time
events
    .keyBy(Event::getUserId)
    .window(TumblingEventTimeWindows.of(Time.minutes(5)))
    .aggregate(new SumAggregator())
    .print();

// Sliding window
events
    .keyBy(Event::getUserId)
    .window(SlidingEventTimeWindows.of(Time.minutes(10), Time.minutes(5)))
    .aggregate(new SumAggregator())
    .print();

// Session windows - Flink's strength
events
    .keyBy(Event::getUserId)
    .window(EventTimeSessionWindows.withGap(Time.minutes(30)))
    .aggregate(new SessionAggregator())
    .print();

Flink’s session windows deserve special mention—they dynamically create windows based on activity gaps, something that’s awkward to implement in Spark. For exactly-once semantics, both frameworks support it, but Flink’s implementation is more battle-tested in high-throughput streaming scenarios.

Batch Processing Performance

Spark’s batch processing remains its strongest suit. Years of optimization, adaptive query execution, and the Catalyst optimizer make it exceptionally efficient for large-scale ETL and analytics workloads.

# Spark - Batch ETL
from pyspark.sql.functions import col, lower, split, explode

# Classic word count demonstrating Spark's batch optimization
text_df = spark.read.text("hdfs:///data/documents/*.txt")

word_counts = text_df \
    .select(explode(split(lower(col("value")), "\\s+")).alias("word")) \
    .filter(col("word") != "") \
    .groupBy("word") \
    .count() \
    .orderBy(col("count").desc())

word_counts.write.parquet("hdfs:///output/word_counts")

// Flink - Batch (DataSet API, though DataStream is now preferred)
import org.apache.flink.api.java.ExecutionEnvironment;

ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

DataSet<String> text = env.readTextFile("hdfs:///data/documents/");

DataSet<Tuple2<String, Integer>> counts = text
    .flatMap(new Tokenizer())
    .groupBy(0)
    .sum(1);

counts.writeAsCsv("hdfs:///output/word_counts");
env.execute();

In practice, Spark typically outperforms Flink on pure batch workloads by 10-30%, depending on the specific operation. Spark’s adaptive query execution dynamically optimizes execution plans based on runtime statistics—a feature Flink is still developing.

State Management and Fault Tolerance

Flink’s state management is genuinely superior for stateful streaming applications. Its state backends (RocksDB for large state, heap for small state) are production-hardened, and savepoints enable operational flexibility that Spark can’t match.

// Flink - Stateful Processing with Keyed State
public class FraudDetector extends KeyedProcessFunction<String, Transaction, Alert> {
    
    // Managed state - survives failures, scales automatically
    private ValueState<Double> runningTotal;
    private ValueState<Long> transactionCount;
    
    @Override
    public void open(Configuration config) {
        ValueStateDescriptor<Double> totalDesc = 
            new ValueStateDescriptor<>("running-total", Double.class);
        ValueStateDescriptor<Long> countDesc = 
            new ValueStateDescriptor<>("tx-count", Long.class);
        
        runningTotal = getRuntimeContext().getState(totalDesc);
        transactionCount = getRuntimeContext().getState(countDesc);
    }
    
    @Override
    public void processElement(Transaction tx, Context ctx, Collector<Alert> out) 
            throws Exception {
        Double total = runningTotal.value();
        Long count = transactionCount.value();
        
        if (total == null) total = 0.0;
        if (count == null) count = 0L;
        
        total += tx.getAmount();
        count += 1;
        
        // Fraud detection logic
        double average = total / count;
        if (tx.getAmount() > average * 10 && count > 5) {
            out.collect(new Alert(tx.getUserId(), "Suspicious transaction"));
        }
        
        runningTotal.update(total);
        transactionCount.update(count);
    }
}

Spark’s stateful streaming exists but feels bolted on. The mapGroupsWithState and flatMapGroupsWithState operations provide similar functionality, but state size is limited by executor memory, and recovery is tied to the micro-batch model.

Flink’s savepoints let you stop a job, modify code, and restart from exactly where you left off—including schema evolution for state. This operational capability is invaluable for long-running streaming applications.

Ecosystem and Integration

Both frameworks offer SQL interfaces for streaming data:

-- Spark SQL on Streaming Data
CREATE TABLE kafka_events (
    user_id STRING,
    event_type STRING,
    amount DOUBLE,
    event_time TIMESTAMP
) USING kafka
OPTIONS (
    'kafka.bootstrap.servers' = 'localhost:9092',
    'subscribe' = 'events'
);

SELECT 
    window(event_time, '5 minutes') as window,
    user_id,
    SUM(amount) as total
FROM kafka_events
GROUP BY window(event_time, '5 minutes'), user_id;

-- Flink SQL on Streaming Data
CREATE TABLE kafka_events (
    user_id STRING,
    event_type STRING,
    amount DOUBLE,
    event_time TIMESTAMP(3),
    WATERMARK FOR event_time AS event_time - INTERVAL '10' SECOND
) WITH (
    'connector' = 'kafka',
    'topic' = 'events',
    'properties.bootstrap.servers' = 'localhost:9092',
    'format' = 'json'
);

SELECT 
    TUMBLE_START(event_time, INTERVAL '5' MINUTE) as window_start,
    user_id,
    SUM(amount) as total
FROM kafka_events
GROUP BY TUMBLE(event_time, INTERVAL '5' MINUTE), user_id;

Spark’s ecosystem is broader. MLlib is more mature than Flink ML. GraphX has no real Flink equivalent. The Databricks platform provides enterprise features that Flink lacks. Spark integrates seamlessly with Delta Lake, while Flink’s Iceberg and Hudi support is catching up.

Flink’s Kafka integration is tighter—unsurprising given their shared streaming DNA. For Kafka-centric architectures, Flink often feels more natural.

When to Choose Which

Choose Spark when:

  • Your workload is 70%+ batch processing
  • You’re already invested in the Databricks/Hadoop ecosystem
  • Your team has Spark experience
  • You need mature ML pipelines
  • Latency requirements are measured in seconds, not milliseconds

Choose Flink when:

  • Real-time processing with sub-second latency is critical
  • You’re building event-driven architectures
  • Complex event processing or pattern matching is required
  • Your streaming jobs maintain large, long-lived state
  • You need session windows or sophisticated event-time processing

The honest truth: most organizations can succeed with either framework. The differences matter at scale and at the margins. If you’re processing a few terabytes daily with relaxed latency requirements, pick the one your team knows. If you’re building a real-time fraud detection system processing millions of events per second with complex stateful logic, Flink’s architecture will serve you better. If you’re running massive ETL pipelines feeding a data warehouse, Spark’s maturity and optimization will pay dividends.

Liked this? There's more.

Every week: one practical technique, explained simply, with code you can use immediately.