How to Handle Null Values in PySpark

Key Insights

Null handling in PySpark requires understanding how nulls propagate through operations—a single unhandled null can silently corrupt aggregations, joins, and comparisons across your entire dataset.
Use na.drop() and na.fill() strategically based on your data’s semantics; dropping rows with any null is rarely the right choice for real-world data pipelines.
Null-safe operations like eqNullSafe() and coalesce() are essential for joins and conditional logic—standard equality comparisons treat null as unknown, not equal.

Introduction

Null values are inevitable in distributed data processing. They creep in from failed API calls, optional form fields, schema mismatches during data ingestion, and outer joins that don’t find matches. In PySpark, mishandling nulls doesn’t throw errors—it silently produces wrong results.

Unlike pandas, where NaN and None have somewhat predictable behavior, PySpark follows SQL semantics where null represents “unknown.” This means null == null returns null, not True. This distinction catches even experienced engineers off guard and leads to subtle bugs that only surface when someone notices the numbers don’t add up.

This article covers the practical techniques you need to detect, filter, fill, and safely operate on null values in PySpark DataFrames.

Detecting Null Values

Before you can handle nulls, you need to know where they are. PySpark provides isNull() and isNotNull() methods on columns for row-level checks, but you’ll often want a summary view across your entire DataFrame.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, count, when, sum as spark_sum

spark = SparkSession.builder.appName("NullHandling").getOrCreate()

# Sample data with nulls
data = [
    (1, "Alice", 50000.0, "Engineering"),
    (2, "Bob", None, "Sales"),
    (3, None, 60000.0, None),
    (4, "Diana", 75000.0, "Engineering"),
    (5, "Eve", None, None),
]
df = spark.createDataFrame(data, ["id", "name", "salary", "department"])

def null_summary(df):
    """Generate null count and percentage for each column."""
    total_rows = df.count()
    null_counts = df.select([
        spark_sum(when(col(c).isNull(), 1).otherwise(0)).alias(c)
        for c in df.columns
    ])
    
    # Convert to more readable format
    summary_data = []
    null_row = null_counts.collect()[0]
    for c in df.columns:
        null_count = null_row[c]
        pct = (null_count / total_rows) * 100
        summary_data.append((c, null_count, round(pct, 2)))
    
    return spark.createDataFrame(summary_data, ["column", "null_count", "null_pct"])

null_summary(df).show()

This produces a summary showing which columns have nulls and their prevalence. Run this early in your pipeline to understand your data quality before making decisions about how to handle missing values.

For quick row-level filtering:

# Rows where salary is null
df.filter(col("salary").isNull()).show()

# Rows where salary is not null
df.filter(col("salary").isNotNull()).show()

Filtering Null Values

Sometimes the right approach is removing rows with nulls entirely. PySpark’s na.drop() method (aliased as dropna()) provides flexible options for this.

# Drop rows where ANY column is null
df_no_nulls = df.na.drop()

# Drop rows where ALL columns are null
df_not_empty = df.na.drop(how="all")

# Drop rows with nulls only in specific columns
df_valid_salary = df.na.drop(subset=["salary"])

# Drop rows with more than 2 null values (thresh = minimum non-null values)
# thresh=3 means keep rows with at least 3 non-null values
df_mostly_complete = df.na.drop(thresh=3)

# Combining subset and thresh
# Keep rows where at least 1 of the specified columns is non-null
df_has_contact = df.na.drop(thresh=1, subset=["name", "department"])

The thresh parameter is often misunderstood. It specifies the minimum number of non-null values required to keep a row. For a DataFrame with 4 columns, thresh=3 keeps rows with at most 1 null value.

# Practical example: keep rows with complete identity info
# but allow missing salary
df_valid_employees = df.na.drop(subset=["id", "name"])
df_valid_employees.show()

Be careful with na.drop() on wide DataFrames—dropping rows with any null can eliminate most of your data.

Filling and Replacing Null Values

When dropping isn’t appropriate, you’ll want to replace nulls with meaningful values. The na.fill() method handles this with type-aware behavior.

# Fill all numeric nulls with 0
df_filled = df.na.fill(0)

# Fill all string nulls with "Unknown"
df_filled = df.na.fill("Unknown")

# Fill specific columns with specific values
df_filled = df.na.fill({
    "salary": 0.0,
    "department": "Unassigned",
    "name": "Anonymous"
})
df_filled.show()

For more sophisticated imputation, you’ll need to compute replacement values from your data:

from pyspark.sql.functions import mean, median, lit

# Fill nulls with column mean
avg_salary = df.select(mean("salary")).collect()[0][0]
df_mean_filled = df.na.fill({"salary": avg_salary})

# Fill with median (requires percentile_approx for large datasets)
from pyspark.sql.functions import percentile_approx

median_salary = df.select(
    percentile_approx("salary", 0.5)
).collect()[0][0]
df_median_filled = df.na.fill({"salary": median_salary})

# Fill with group-specific values (e.g., department average)
from pyspark.sql.window import Window

dept_avg = df.groupBy("department").agg(
    mean("salary").alias("dept_avg_salary")
)

df_with_avg = df.join(dept_avg, "department", "left")
df_imputed = df_with_avg.withColumn(
    "salary",
    when(col("salary").isNull(), col("dept_avg_salary"))
    .otherwise(col("salary"))
).drop("dept_avg_salary")

Choose your fill strategy based on your data’s semantics. Filling numeric nulls with 0 makes sense for counts but distorts salary averages. Mean imputation preserves distribution center but reduces variance.

Null-Safe Operations

Standard comparisons in PySpark follow SQL’s three-valued logic: comparisons involving null return null, not False. This breaks intuitive expectations.

# This does NOT work as expected
df.filter(col("department") == None).show()  # Returns empty!

# This is correct
df.filter(col("department").isNull()).show()

# Null comparisons return null, not False
df.select(
    col("department"),
    (col("department") == "Sales").alias("is_sales"),
    (col("department") == None).alias("is_none_wrong")
).show()

For null-safe equality in joins, use eqNullSafe():

# Create two DataFrames with potential null join keys
employees = spark.createDataFrame([
    (1, "Alice", "A"),
    (2, "Bob", None),
    (3, "Charlie", "B"),
], ["id", "name", "dept_code"])

departments = spark.createDataFrame([
    ("A", "Engineering"),
    ("B", "Sales"),
    (None, "Unassigned"),
], ["code", "dept_name"])

# Standard join misses null matches
standard_join = employees.join(
    departments,
    employees.dept_code == departments.code,
    "left"
)
standard_join.show()  # Bob's dept_name is null

# Null-safe join matches null to null
nullsafe_join = employees.join(
    departments,
    employees.dept_code.eqNullSafe(departments.code),
    "left"
)
nullsafe_join.show()  # Bob gets "Unassigned"

The coalesce() function returns the first non-null value from a list of columns—essential for providing fallbacks:

from pyspark.sql.functions import coalesce

# Use department if available, otherwise "General"
df_with_fallback = df.withColumn(
    "effective_dept",
    coalesce(col("department"), lit("General"))
)

# Chain multiple fallbacks
df_multi = df.withColumn(
    "display_name",
    coalesce(col("name"), col("department"), lit("Unknown"))
)

Handling Nulls in Aggregations

Aggregation functions in PySpark generally ignore nulls, but the behavior varies:

from pyspark.sql.functions import count, sum as spark_sum, avg

# count(*) counts all rows; count(col) counts non-null values
df.select(
    count("*").alias("total_rows"),
    count("salary").alias("non_null_salaries"),
    spark_sum("salary").alias("total_salary"),
    avg("salary").alias("avg_salary")
).show()

With our sample data, count("*") returns 5, but count("salary") returns 3 (only non-null values). The avg() function divides by the count of non-null values, not total rows.

Window functions have an ignorenulls parameter for functions like first() and last():

from pyspark.sql.window import Window
from pyspark.sql.functions import first, last

window_spec = Window.partitionBy("department").orderBy("id")

# Get first non-null salary in department
df_with_first = df.withColumn(
    "first_dept_salary",
    first("salary", ignorenulls=True).over(window_spec)
)

# Forward-fill nulls with last known value
df_ffill = df.withColumn(
    "salary_ffill",
    last("salary", ignorenulls=True).over(
        Window.partitionBy("department")
        .orderBy("id")
        .rowsBetween(Window.unboundedPreceding, Window.currentRow)
    )
)

Best Practices and Performance Considerations

Handle nulls early when you have clear semantics. If you know that null salaries should be 0 or that missing departments should be “Unknown,” apply those transformations at ingestion time. This prevents repeated null checks throughout your pipeline.

Handle nulls late when semantics are context-dependent. If a null salary means “not disclosed” in one report but “unpaid intern” in another, preserve the nulls and handle them at the point of use.

Avoid repeated null checks on the same column. Each isNull() check adds computation. If you’re filtering and then filling, consider whether you can restructure:

# Inefficient: checking null twice
df_filtered = df.filter(col("salary").isNotNull())
df_filled = df.filter(col("salary").isNull()).na.fill({"salary": 0})
df_combined = df_filtered.union(df_filled)

# Better: single pass with when/otherwise
df_handled = df.withColumn(
    "salary",
    when(col("salary").isNull(), 0).otherwise(col("salary"))
)

Design schemas to minimize nulls. Use default values in upstream systems when possible. Separate optional fields into their own tables rather than having wide tables with many nullable columns.

Document your null semantics. When you fill nulls with sentinel values like -1 or “Unknown,” document this clearly. Future consumers of your data need to know that -1 doesn’t mean a negative value but rather missing data.

Test with null-heavy data. Create test datasets that stress your null handling—rows with all nulls, columns with all nulls, and the edge cases your production data will eventually contain.

Null handling isn’t glamorous, but it’s the difference between a pipeline that works and one that produces subtly wrong results. Get it right early, and you’ll save hours of debugging later.