PySpark - Rename Multiple Columns | Application Architect

Key Insights

Use dictionary-based renaming with reduce() or list comprehensions for renaming multiple columns efficiently—it’s more maintainable than chaining withColumnRenamed() calls
The toDF() method is fastest for renaming all columns but requires you to specify every column name in the correct order, making it error-prone for wide DataFrames
Leverage selectExpr() with f-strings for pattern-based transformations like adding prefixes or converting to snake_case, enabling bulk column standardization with minimal code

Introduction

Column renaming is one of the most common data preparation tasks in PySpark. Whether you’re standardizing column names across datasets for joins, cleaning up messy source data, or conforming to your organization’s naming conventions, you’ll need efficient ways to rename multiple columns.

PySpark offers several approaches for renaming columns, each with different trade-offs in terms of readability, performance, and flexibility. Choosing the right method depends on how many columns you’re renaming, whether you need pattern-based transformations, and how you want to handle the rest of your DataFrame.

Let’s start with a sample DataFrame that demonstrates typical naming problems you might encounter:

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType

spark = SparkSession.builder.appName("RenameColumns").getOrCreate()

# Sample DataFrame with poorly named columns
data = [
    ("John", 28, "New York", 75000),
    ("Sarah", 34, "San Francisco", 95000),
    ("Mike", 45, "Chicago", 82000)
]

df = spark.createDataFrame(data, ["Name", "age_Years", "City of Residence", "Annual Salary ($)"])
df.show()

This DataFrame has inconsistent naming: mixed case, spaces, special characters, and unclear abbreviations. Let’s explore how to fix this systematically.

Using `withColumnRenamed()` Method

The withColumnRenamed() method is PySpark’s most straightforward approach for renaming columns. It takes two arguments: the existing column name and the new name. For renaming a small number of columns, chaining multiple calls is perfectly acceptable.

df_renamed = (df
    .withColumnRenamed("Name", "employee_name")
    .withColumnRenamed("age_Years", "age")
    .withColumnRenamed("City of Residence", "city")
)

df_renamed.show()

This approach is explicit and readable, making it ideal when you’re renaming just a few columns or when the renaming logic needs to be obvious to other developers. However, chaining becomes unwieldy when dealing with dozens of columns. Each withColumnRenamed() call returns a new DataFrame, which can impact performance if you’re not careful, though Spark’s optimizer typically handles this well.

The main limitation is maintainability: a chain of 20 withColumnRenamed() calls is harder to review and modify than a data structure that maps old names to new ones.

Using `toDF()` Method

The toDF() method renames all columns in a single operation by accepting a list of new column names. This is the most performant option when you need to rename every column.

new_column_names = ["employee_name", "age", "city", "annual_salary"]
df_renamed = df.toDF(*new_column_names)

df_renamed.show()

The asterisk (*) unpacks the list as individual arguments to toDF(). This method is fast and concise, but it has a critical requirement: you must provide names for all columns in the exact order they appear in the DataFrame.

This makes toDF() error-prone for wide DataFrames with many columns. If your DataFrame has 50 columns and you only want to rename 5, you still need to specify all 50 names. Missing a column or getting the order wrong will misalign your data.

Use toDF() when you’re either creating a new DataFrame from scratch or when you genuinely need to rename every single column.

Using `select()` with `alias()`

The select() method combined with alias() gives you fine-grained control over which columns to rename and which to keep as-is. This is particularly useful when renaming a subset of columns.

from pyspark.sql.functions import col

df_renamed = df.select(
    col("Name").alias("employee_name"),
    col("age_Years").alias("age"),
    col("City of Residence").alias("city"),
    col("Annual Salary ($)")  # Keep original name
)

df_renamed.show()

This approach explicitly shows which columns you’re working with, making it excellent for documentation purposes. You can also perform transformations while renaming:

from pyspark.sql.functions import lower

df_renamed = df.select(
    lower(col("Name")).alias("employee_name"),
    col("age_Years").alias("age"),
    col("City of Residence").alias("city"),
    col("Annual Salary ($)").alias("annual_salary")
)

The downside is verbosity when dealing with many columns. If you have 30 columns and only need to rename 3, you still need to explicitly list all 30 in your select() statement.

Using Dictionary Mapping with Loop

For renaming multiple columns efficiently, a dictionary mapping old names to new names provides the best balance of readability and maintainability. You can combine this with reduce() for a functional approach or use a simple loop.

from functools import reduce

# Define your column mappings
column_mapping = {
    "Name": "employee_name",
    "age_Years": "age",
    "City of Residence": "city",
    "Annual Salary ($)": "annual_salary"
}

# Functional approach using reduce
df_renamed = reduce(
    lambda df, col_name: df.withColumnRenamed(col_name, column_mapping[col_name]),
    column_mapping.keys(),
    df
)

df_renamed.show()

Alternatively, use a traditional loop for better readability:

df_renamed = df
for old_name, new_name in column_mapping.items():
    df_renamed = df_renamed.withColumnRenamed(old_name, new_name)

df_renamed.show()

This approach scales well because your mapping is a clear data structure that’s easy to modify, review, and even load from configuration files. You can also handle columns that don’t exist gracefully:

df_renamed = df
for old_name, new_name in column_mapping.items():
    if old_name in df.columns:
        df_renamed = df_renamed.withColumnRenamed(old_name, new_name)

Using `selectExpr()` for Pattern-Based Renaming

When you need to apply systematic transformations to column names—like adding prefixes, removing suffixes, or converting to snake_case—selectExpr() with list comprehensions provides an elegant solution.

# Convert all column names to lowercase
df_lowercase = df.selectExpr(*[f"`{col}` as {col.lower().replace(' ', '_')}" for col in df.columns])
df_lowercase.show()

The backticks around the original column name handle columns with spaces or special characters. Here’s a more sophisticated example that adds a prefix:

# Add 'customer_' prefix to all columns
prefix = "customer_"
df_prefixed = df.selectExpr(*[f"`{col}` as {prefix}{col.lower().replace(' ', '_')}" for col in df.columns])
df_prefixed.show()

You can also combine pattern-based renaming with selective logic:

# Convert to snake_case, but only for specific columns
def to_snake_case(name):
    return name.lower().replace(' ', '_').replace('(', '').replace(')', '').replace('$', 'usd')

columns_to_rename = ["City of Residence", "Annual Salary ($)"]

select_expr = []
for col in df.columns:
    if col in columns_to_rename:
        select_expr.append(f"`{col}` as {to_snake_case(col)}")
    else:
        select_expr.append(f"`{col}`")

df_renamed = df.selectExpr(*select_expr)
df_renamed.show()

This approach shines when you’re dealing with datasets where column names follow predictable patterns that need systematic cleanup.

Performance Considerations & Best Practices

All these renaming methods are metadata operations in Spark—they don’t trigger data movement or computation until an action is called. However, some best practices will make your code more maintainable and performant.

Rename early in your pipeline: Apply column renaming immediately after reading data, before expensive transformations. This makes your subsequent code cleaner and prevents confusion about which column names are current.

# Good: Rename immediately after reading
df = spark.read.csv("data.csv", header=True)
df = df.toDF(*standardized_column_names)
# ... rest of transformations use clean names

# Less ideal: Rename after complex transformations
df = spark.read.csv("data.csv", header=True)
# ... many transformations using original messy names
df = df.toDF(*standardized_column_names)  # Too late

Choose the right method for your use case:

1-3 columns: Chain withColumnRenamed() for clarity
All columns: Use toDF() for performance
Subset with transformations: Use select() with alias()
Many columns with mapping: Use dictionary with loop or reduce()
Pattern-based bulk changes: Use selectExpr() with list comprehensions

Avoid renaming in loops over rows: Never rename columns inside operations that process individual rows. Column renaming is a schema operation and should happen at the DataFrame level.

Validate column existence: When using dictionary mappings, check that columns exist before renaming to avoid runtime errors in production:

def safe_rename(df, column_mapping):
    for old_name, new_name in column_mapping.items():
        if old_name in df.columns:
            df = df.withColumnRenamed(old_name, new_name)
        else:
            print(f"Warning: Column '{old_name}' not found in DataFrame")
    return df

Document your naming conventions: Maintain a clear standard for column names (snake_case, camelCase, etc.) and apply it consistently across your organization’s data pipelines.

The most maintainable approach for production systems is typically a dictionary mapping combined with a helper function that handles validation and logging. This gives you the flexibility to manage complex renaming logic while keeping your code testable and debuggable.

Introduction

Using withColumnRenamed() Method

Using toDF() Method

Using select() with alias()