PySpark - RDD vs DataFrame - When to Use Which
• RDDs provide low-level control and are essential for unstructured data or custom partitioning logic, but lack automatic optimization and require manual schema management
Key Insights
• RDDs provide low-level control and are essential for unstructured data or custom partitioning logic, but lack automatic optimization and require manual schema management • DataFrames offer Catalyst optimizer benefits, built-in operations, and better performance for structured data through predicate pushdown and columnar storage • Choose RDDs when you need fine-grained control over data partitioning or are working with non-tabular data; choose DataFrames for 90% of structured data processing tasks
Understanding the Fundamental Difference
RDDs (Resilient Distributed Datasets) represent PySpark’s low-level API, providing immutable distributed collections of objects. DataFrames build on top of RDDs, adding structure and optimization through the Catalyst query optimizer and Tungsten execution engine.
from pyspark.sql import SparkSession
from pyspark import SparkContext
spark = SparkSession.builder.appName("RDDvsDF").getOrCreate()
sc = spark.sparkContext
# RDD: Collection of Python objects
rdd_data = sc.parallelize([
("Alice", 25, "Engineering"),
("Bob", 30, "Sales"),
("Charlie", 35, "Engineering")
])
# DataFrame: Structured data with schema
df_data = spark.createDataFrame([
("Alice", 25, "Engineering"),
("Bob", 30, "Sales"),
("Charlie", 35, "Engineering")
], ["name", "age", "department"])
The RDD stores tuples as Python objects, while the DataFrame understands the schema and stores data in optimized columnar format. This structural awareness enables significant performance improvements.
Performance Characteristics
DataFrames consistently outperform RDDs for structured data operations due to Catalyst optimization and Tungsten’s code generation.
import time
# RDD approach - no optimization
def rdd_filter_aggregate():
start = time.time()
result = (rdd_data
.filter(lambda x: x[2] == "Engineering")
.map(lambda x: (x[2], x[1]))
.reduceByKey(lambda a, b: a + b)
.collect())
return time.time() - start, result
# DataFrame approach - optimized execution plan
def df_filter_aggregate():
start = time.time()
result = (df_data
.filter(df_data.department == "Engineering")
.groupBy("department")
.sum("age")
.collect())
return time.time() - start, result
# With larger datasets, DataFrame will be significantly faster
large_rdd = sc.parallelize([(f"Person{i}", i % 100, "Dept" + str(i % 5))
for i in range(1000000)])
large_df = spark.createDataFrame(large_rdd, ["name", "age", "department"])
The DataFrame version benefits from predicate pushdown, which filters data before aggregation. The Catalyst optimizer reorders operations for efficiency, while RDDs execute operations in the exact order specified.
When RDDs Are the Right Choice
Custom Partitioning Logic
RDDs excel when you need precise control over data partitioning across the cluster.
from pyspark import Partitioner
class CustomPartitioner(Partitioner):
def __init__(self, num_partitions):
self._num_partitions = num_partitions
def numPartitions(self):
return self._num_partitions
def getPartition(self, key):
# Custom logic: route specific keys to specific partitions
if key.startswith('A'):
return 0
elif key.startswith('B'):
return 1
else:
return hash(key) % self._num_partitions
# Apply custom partitioning
key_value_rdd = sc.parallelize([
("Alice", {"age": 25, "dept": "Eng"}),
("Bob", {"age": 30, "dept": "Sales"}),
("Andrew", {"age": 28, "dept": "Eng"}),
("Barbara", {"age": 32, "dept": "HR"})
])
partitioned_rdd = key_value_rdd.partitionBy(3, CustomPartitioner(3))
# Verify partitioning
def show_partition_distribution(rdd):
return rdd.mapPartitionsWithIndex(
lambda idx, it: [(idx, list(it))]
).collect()
print(show_partition_distribution(partitioned_rdd))
DataFrames don’t support custom partitioners. While you can use repartition() or partitionBy() for bucketing, you can’t implement complex partitioning logic.
Unstructured or Semi-Structured Data
When processing data that doesn’t fit tabular structures, RDDs provide necessary flexibility.
# Processing raw text logs with complex patterns
log_rdd = sc.textFile("application.log")
def parse_complex_log(line):
# Custom parsing logic for non-standard format
if line.startswith("ERROR"):
parts = line.split("|")
return {
"level": "ERROR",
"timestamp": parts[1] if len(parts) > 1 else None,
"message": parts[2] if len(parts) > 2 else None,
"stack_trace": parts[3:] if len(parts) > 3 else []
}
return None
parsed_logs = log_rdd.map(parse_complex_log).filter(lambda x: x is not None)
# Group errors by hour
errors_by_hour = (parsed_logs
.map(lambda x: (x["timestamp"][:13], 1))
.reduceByKey(lambda a, b: a + b)
.collect())
When DataFrames Are the Right Choice
Structured Data Operations
For any tabular data with consistent schema, DataFrames provide superior performance and cleaner syntax.
from pyspark.sql import functions as F
# Read structured data
sales_df = spark.read.parquet("sales_data.parquet")
# Complex aggregations with optimization
result = (sales_df
.filter(F.col("sale_date") >= "2024-01-01")
.groupBy("region", "product_category")
.agg(
F.sum("revenue").alias("total_revenue"),
F.avg("quantity").alias("avg_quantity"),
F.countDistinct("customer_id").alias("unique_customers")
)
.withColumn("revenue_per_customer",
F.col("total_revenue") / F.col("unique_customers"))
.orderBy(F.desc("total_revenue")))
# Catalyst optimizer handles:
# - Predicate pushdown (filter before aggregation)
# - Column pruning (only read needed columns)
# - Optimal join strategies
SQL Integration and Analytics
DataFrames seamlessly integrate with SQL, enabling familiar query patterns and integration with BI tools.
# Register DataFrame as temporary view
sales_df.createOrReplaceTempView("sales")
# Mix SQL and DataFrame API
sql_result = spark.sql("""
SELECT
region,
product_category,
SUM(revenue) as total_revenue,
RANK() OVER (PARTITION BY region ORDER BY SUM(revenue) DESC) as rank
FROM sales
WHERE sale_date >= '2024-01-01'
GROUP BY region, product_category
""")
# Continue with DataFrame operations
top_performers = (sql_result
.filter(F.col("rank") <= 3)
.select("region", "product_category", "total_revenue"))
Converting Between RDDs and DataFrames
Sometimes you need to leverage both APIs in the same pipeline.
# DataFrame to RDD
df_to_rdd = df_data.rdd.map(lambda row: (row.name, row.age))
# RDD to DataFrame - explicit schema
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
schema = StructType([
StructField("name", StringType(), True),
StructField("age", IntegerType(), True),
StructField("department", StringType(), True)
])
rdd_to_df = spark.createDataFrame(rdd_data, schema)
# RDD to DataFrame - inferred schema (slower)
rdd_to_df_inferred = rdd_data.toDF(["name", "age", "department"])
Memory and Serialization Considerations
RDDs use Java/Python object serialization, while DataFrames use optimized binary format.
# RDD: Each object serialized individually
class Employee:
def __init__(self, name, age, dept):
self.name = name
self.age = age
self.dept = dept
employee_rdd = sc.parallelize([
Employee("Alice", 25, "Eng"),
Employee("Bob", 30, "Sales")
])
# Heavy serialization overhead for custom objects
# DataFrame: Columnar storage, efficient compression
employee_df = spark.createDataFrame([
("Alice", 25, "Eng"),
("Bob", 30, "Sales")
], ["name", "age", "dept"])
# Check physical plan
employee_df.explain(mode="formatted")
Practical Decision Framework
Use RDDs when:
- Implementing custom partitioning strategies for specific data locality requirements
- Processing unstructured data that doesn’t fit tabular models
- Requiring fine-grained control over transformation logic
- Working with complex Python objects that don’t serialize well to DataFrames
Use DataFrames when:
- Processing structured or semi-structured data (JSON, Parquet, CSV)
- Performing standard analytical operations (aggregations, joins, window functions)
- Requiring SQL compatibility or integration with BI tools
- Performance is critical and data fits tabular structure
- Working with large datasets where Catalyst optimization provides significant benefits
The industry trend strongly favors DataFrames for new development. RDDs remain relevant for specialized use cases requiring low-level control, but DataFrames should be your default choice for structured data processing tasks.