Eric Campbell 6 min read Pandas

How to Use Pipe in Pandas

If you've written Pandas code for any length of time, you've probably encountered the readability nightmare of nested function calls or sprawling intermediate variables. The `pipe()` method solves...

Key Insights

  • The pipe() method enables clean, readable data pipelines by passing entire DataFrames through custom functions, making complex transformations easier to understand and maintain.
  • Unlike apply() which operates row-by-row or column-by-column, pipe() works on the whole DataFrame, making it ideal for multi-step transformations that need full data context.
  • Writing small, pure functions designed for piping creates reusable building blocks that you can compose into sophisticated data workflows without sacrificing readability.

Introduction to the Pipe Method

If you’ve written Pandas code for any length of time, you’ve probably encountered the readability nightmare of nested function calls or sprawling intermediate variables. The pipe() method solves this problem elegantly by enabling method chaining with custom functions.

Method chaining isn’t just about aesthetics. When you chain operations, you create a clear narrative of data transformations that reads top-to-bottom. Anyone reviewing your code can follow the logic without mentally unwinding nested parentheses or tracking variable reassignments across dozens of lines.

The pipe() method was added to Pandas specifically to let you incorporate your own functions into these chains. It’s the bridge between Pandas’ built-in methods and your custom logic.

Basic Syntax and How Pipe Works

The pipe() method has a straightforward signature:

DataFrame.pipe(func, *args, **kwargs)

When you call pipe(), Pandas passes the DataFrame as the first argument to whatever function you provide. Your function receives the DataFrame, does its work, and returns a DataFrame (or Series) that continues down the chain.

Here’s a simple example:

import pandas as pd

def add_full_name(df):
    """Combine first and last name into a full_name column."""
    df = df.copy()
    df['full_name'] = df['first_name'] + ' ' + df['last_name']
    return df

# Sample data
employees = pd.DataFrame({
    'first_name': ['Alice', 'Bob', 'Carol'],
    'last_name': ['Smith', 'Jones', 'Williams'],
    'salary': [75000, 82000, 91000]
})

# Using pipe
result = employees.pipe(add_full_name)
print(result)

Output:

  first_name last_name  salary        full_name
0      Alice     Smith   75000      Alice Smith
1        Bob     Jones   82000        Bob Jones
2      Carol  Williams   91000  Carol Williams

Notice that add_full_name creates a copy before modifying. This keeps the function pure—it doesn’t mutate the input DataFrame. We’ll revisit why this matters later.

Chaining Multiple Operations

The real power of pipe() emerges when you chain multiple operations together. Each step flows naturally into the next, creating a readable pipeline.

import pandas as pd
import numpy as np

# Sample sales data
sales = pd.DataFrame({
    'date': pd.date_range('2024-01-01', periods=100, freq='D'),
    'product': np.random.choice(['Widget', 'Gadget', 'Gizmo'], 100),
    'quantity': np.random.randint(1, 50, 100),
    'unit_price': np.random.uniform(10, 100, 100).round(2),
    'region': np.random.choice(['North', 'South', 'East', 'West'], 100)
})

def filter_by_region(df, regions):
    """Keep only specified regions."""
    return df[df['region'].isin(regions)]

def calculate_revenue(df):
    """Add revenue column."""
    df = df.copy()
    df['revenue'] = df['quantity'] * df['unit_price']
    return df

def aggregate_by_product(df):
    """Summarize by product."""
    return df.groupby('product').agg({
        'quantity': 'sum',
        'revenue': 'sum'
    }).reset_index()

# Build the pipeline
summary = (
    sales
    .pipe(filter_by_region, regions=['North', 'East'])
    .pipe(calculate_revenue)
    .pipe(aggregate_by_product)
)

print(summary)

Read this code from top to bottom: take sales data, filter to North and East regions, calculate revenue, then aggregate by product. The intent is immediately clear. Compare this to the alternative:

# Without pipe - harder to follow
summary = aggregate_by_product(
    calculate_revenue(
        filter_by_region(sales, regions=['North', 'East'])
    )
)

The nested version requires reading inside-out, which becomes increasingly painful as pipelines grow.

Passing Additional Arguments

You’ve already seen one way to pass arguments—simply include them after the function name. Pandas forwards any additional positional or keyword arguments to your function.

def filter_by_threshold(df, column, threshold, keep='above'):
    """
    Filter rows based on a numeric threshold.
    
    Parameters:
    - column: Column name to filter on
    - threshold: Numeric threshold value
    - keep: 'above' or 'below' to specify which rows to keep
    """
    if keep == 'above':
        return df[df[column] > threshold]
    elif keep == 'below':
        return df[df[column] < threshold]
    else:
        raise ValueError("keep must be 'above' or 'below'")

def cap_outliers(df, column, lower_percentile=5, upper_percentile=95):
    """Cap values at specified percentiles."""
    df = df.copy()
    lower = df[column].quantile(lower_percentile / 100)
    upper = df[column].quantile(upper_percentile / 100)
    df[column] = df[column].clip(lower, upper)
    return df

# Using with different configurations
result = (
    sales
    .pipe(filter_by_threshold, 'quantity', 10, keep='above')
    .pipe(cap_outliers, 'unit_price', lower_percentile=10, upper_percentile=90)
)

This pattern creates reusable functions that adapt to different contexts. The same filter_by_threshold function works for any column and any threshold value.

Pipe vs. Apply vs. Transform

New Pandas users often confuse pipe(), apply(), and transform(). Here’s the distinction:

  • pipe(): Operates on the entire DataFrame or Series. Your function receives the whole object.
  • apply(): Operates row-by-row or column-by-column. Your function receives individual rows or columns.
  • transform(): Like apply(), but must return output with the same shape as input.
df = pd.DataFrame({
    'A': [1, 2, 3, 4, 5],
    'B': [10, 20, 30, 40, 50]
})

# PIPE: Function receives entire DataFrame
def add_ratio_column(df):
    df = df.copy()
    df['ratio'] = df['A'] / df['B']
    return df

result_pipe = df.pipe(add_ratio_column)

# APPLY: Function receives each row (axis=1) or column (axis=0)
def row_sum(row):
    return row['A'] + row['B']

result_apply = df.apply(row_sum, axis=1)  # Returns Series

# TRANSFORM: Function receives each column, must return same shape
def standardize(col):
    return (col - col.mean()) / col.std()

result_transform = df.transform(standardize)

print("Pipe result:")
print(result_pipe)
print("\nApply result (row sums):")
print(result_apply)
print("\nTransform result (standardized):")
print(result_transform)

Use pipe() when your operation needs context from the entire DataFrame—like calculating ratios between columns, filtering based on aggregate statistics, or adding multiple derived columns. Use apply() when you need to process rows or columns independently.

Real-World Use Case: Building a Data Cleaning Pipeline

Let’s build a complete data cleaning pipeline that handles messy real-world data:

import pandas as pd
import numpy as np

# Messy raw data
raw_data = pd.DataFrame({
    'order_id': ['001', '002', '003', '004', '005', '006'],
    'customer_name': ['john doe', 'JANE SMITH', '  Bob Wilson  ', 'alice brown', None, 'charlie davis'],
    'order_date': ['2024-01-15', '2024-01-16', 'invalid', '2024-01-18', '2024-01-19', '2024-01-20'],
    'amount': [150.00, -50.00, 200.00, 10000.00, 175.00, 225.00],  # -50 is error, 10000 is outlier
    'status': ['completed', 'completed', 'pending', 'completed', 'completed', 'shipped']
})

def clean_customer_names(df):
    """Standardize customer name formatting."""
    df = df.copy()
    df['customer_name'] = (
        df['customer_name']
        .str.strip()
        .str.title()
        .fillna('Unknown Customer')
    )
    return df

def parse_dates(df, date_column):
    """Convert date strings to datetime, coercing errors to NaT."""
    df = df.copy()
    df[date_column] = pd.to_datetime(df[date_column], errors='coerce')
    return df

def remove_invalid_amounts(df, amount_column):
    """Remove rows with negative or zero amounts."""
    return df[df[amount_column] > 0]

def cap_amount_outliers(df, amount_column, max_multiple=10):
    """Cap amounts at max_multiple times the median."""
    df = df.copy()
    median_amount = df[amount_column].median()
    cap_value = median_amount * max_multiple
    df[amount_column] = df[amount_column].clip(upper=cap_value)
    return df

def add_derived_features(df):
    """Add useful derived columns."""
    df = df.copy()
    df['order_month'] = df['order_date'].dt.to_period('M')
    df['is_high_value'] = df['amount'] > df['amount'].median()
    return df

def drop_incomplete_records(df, required_columns):
    """Remove rows with missing values in required columns."""
    return df.dropna(subset=required_columns)

# The complete pipeline
cleaned_data = (
    raw_data
    .pipe(clean_customer_names)
    .pipe(parse_dates, 'order_date')
    .pipe(remove_invalid_amounts, 'amount')
    .pipe(cap_amount_outliers, 'amount', max_multiple=5)
    .pipe(add_derived_features)
    .pipe(drop_incomplete_records, ['order_date', 'customer_name'])
)

print("Cleaned data:")
print(cleaned_data)

This pipeline handles name normalization, date parsing, invalid data removal, outlier capping, feature engineering, and missing value handling—all in a readable, maintainable format.

Best Practices and Tips

Write pure functions. Always return a new DataFrame rather than modifying in place. Use df.copy() at the start of functions that modify data. This prevents subtle bugs where earlier pipeline stages affect later ones unexpectedly.

Keep functions focused. Each piped function should do one thing well. If a function is doing multiple unrelated transformations, split it up. Small functions are easier to test, debug, and reuse.

Name functions clearly. Function names should describe the transformation: remove_duplicates, standardize_column_names, filter_active_users. Future you will thank present you.

Return DataFrames consistently. Every function in a chain should return a DataFrame (or Series if that’s what you’re working with). Functions that return None or other types break the chain.

Don’t overuse pipe. If you’re just calling a single built-in method, don’t wrap it in a function just to use pipe. df.dropna() is clearer than df.pipe(lambda x: x.dropna()). Use pipe when it genuinely improves readability.

Consider logging for debugging. When pipelines get complex, add a logging function:

def log_shape(df, step_name):
    print(f"{step_name}: {df.shape[0]} rows, {df.shape[1]} columns")
    return df

# Insert between steps for debugging
result = (
    df
    .pipe(step_one)
    .pipe(log_shape, 'After step one')
    .pipe(step_two)
    .pipe(log_shape, 'After step two')
)

The pipe() method transforms how you structure Pandas code. It encourages modular, testable functions and produces code that reads like documentation. Once you start thinking in pipelines, you’ll find your data analysis code becomes significantly easier to write, review, and maintain.

Liked this? There's more.

Every week: one practical technique, explained simply, with code you can use immediately.

Similar Articles