How to Handle Missing Data in Python

Key Insights

• Understanding why data is missing (MCAR, MAR, or MNAR) determines which handling strategy will preserve statistical validity—choosing wrong can introduce severe bias
• Simple imputation methods like mean/median work for quick analyses, but machine learning approaches like KNN and MICE produce more accurate results when relationships between variables matter
• Deletion is only safe when missing data is truly random and represents less than 5% of your dataset; otherwise, you’re throwing away information and potentially skewing results

Introduction to Missing Data

Missing data isn’t just an inconvenience—it’s a statistical landmine. Every dataset you encounter in production will have gaps, and how you handle them directly impacts the validity of your analysis. Get it wrong, and you’ll draw conclusions from biased data.

Before reaching for dropna(), you need to understand the mechanism behind the missingness. Statisticians classify missing data into three categories:

Missing Completely at Random (MCAR): The probability of a value being missing is unrelated to any variable in the dataset. A sensor randomly failing due to hardware issues produces MCAR data. This is the best-case scenario.

Missing at Random (MAR): The missingness depends on observed variables but not the missing value itself. Higher-income respondents might skip income questions, but once you control for education level, the missingness is random.

Missing Not at Random (MNAR): The missingness depends on the unobserved value. People with extreme values (very high income, severe symptoms) systematically skip questions. This is the hardest to handle correctly.

Most real-world data falls into MAR territory. Assuming MCAR when it’s actually MNAR will corrupt your analysis.

Detecting Missing Data

Before fixing anything, quantify the problem. Pandas provides straightforward tools for this.

import pandas as pd
import numpy as np

# Load your dataset
df = pd.read_csv('customer_data.csv')

# Quick overview of missing values
print(df.isnull().sum())

# Percentage missing per column
missing_pct = (df.isnull().sum() / len(df)) * 100
print(missing_pct.sort_values(ascending=False))

# DataFrame info shows non-null counts
df.info()

For a more detailed view, check which rows have any missing values:

# Count rows with at least one missing value
rows_with_missing = df.isnull().any(axis=1).sum()
print(f"Rows with missing data: {rows_with_missing} ({rows_with_missing/len(df)*100:.1f}%)")

# Identify patterns in missingness
missing_matrix = df.isnull().astype(int)
print(missing_matrix.corr())  # Correlation between missing patterns

The missingno library provides visualizations that reveal patterns human eyes can catch but summary statistics miss:

import missingno as msno
import matplotlib.pyplot as plt

# Matrix visualization - white lines show missing values
msno.matrix(df)
plt.show()

# Bar chart of completeness
msno.bar(df)
plt.show()

# Heatmap of missing value correlations
msno.heatmap(df)
plt.show()

The heatmap is particularly useful. High correlation between missing values in two columns suggests a systematic pattern—possibly MAR or MNAR data that requires careful handling.

Removal Strategies

Dropping missing data is the simplest approach, but it’s only appropriate under specific conditions: the data is MCAR, the proportion missing is small (under 5%), and you have enough samples to maintain statistical power.

# Drop rows with any missing values
df_clean = df.dropna()

# Drop rows only if specific columns have missing values
df_clean = df.dropna(subset=['age', 'income'])

# Drop rows that have fewer than 3 non-null values
df_clean = df.dropna(thresh=3)

# Drop columns instead of rows (when a column is mostly missing)
df_clean = df.dropna(axis=1, thresh=len(df) * 0.5)  # Keep columns with 50%+ data

A practical pattern for handling columns with extreme missingness:

# Remove columns missing more than 40% of values
threshold = 0.4
cols_to_drop = missing_pct[missing_pct > threshold * 100].index
df_reduced = df.drop(columns=cols_to_drop)
print(f"Dropped {len(cols_to_drop)} columns: {list(cols_to_drop)}")

Warning: If your missing data isn’t MCAR, deletion introduces bias. Dropping all rows with missing income when high earners skip that question means your analysis underestimates average income.

Imputation Techniques

Imputation replaces missing values with estimates. Simple statistical imputation works well for exploratory analysis and when speed matters more than precision.

# Fill with column mean (numeric data)
df['age'] = df['age'].fillna(df['age'].mean())

# Fill with median (better for skewed distributions)
df['income'] = df['income'].fillna(df['income'].median())

# Fill with mode (categorical data)
df['category'] = df['category'].fillna(df['category'].mode()[0])

# Fill with a constant
df['status'] = df['status'].fillna('Unknown')

For production pipelines, use scikit-learn’s SimpleImputer to ensure consistent handling between training and inference:

from sklearn.impute import SimpleImputer

# Numeric columns
num_imputer = SimpleImputer(strategy='median')
df[['age', 'income']] = num_imputer.fit_transform(df[['age', 'income']])

# Categorical columns
cat_imputer = SimpleImputer(strategy='most_frequent')
df[['category', 'region']] = cat_imputer.fit_transform(df[['category', 'region']])

Time series data requires different treatment. Forward fill and backward fill preserve temporal relationships:

# Forward fill - use last known value
df['sensor_reading'] = df['sensor_reading'].ffill()

# Backward fill - use next known value
df['sensor_reading'] = df['sensor_reading'].bfill()

# Interpolation for numeric time series
df['temperature'] = df['temperature'].interpolate(method='linear')

# Time-based interpolation when you have a datetime index
df['value'] = df['value'].interpolate(method='time')

The limitation of simple imputation: it reduces variance and weakens correlations between variables. Filling every missing age with the mean age makes your age distribution artificially peaked.

Advanced Imputation Methods

When relationships between variables matter—and they usually do in statistical modeling—machine learning imputation produces better results.

KNN Imputation finds the k nearest neighbors based on non-missing features and uses their values to impute:

from sklearn.impute import KNNImputer

# KNN uses Euclidean distance, so scale your features first
from sklearn.preprocessing import StandardScaler

# Prepare numeric data
numeric_cols = df.select_dtypes(include=[np.number]).columns
df_numeric = df[numeric_cols].copy()

# Impute with 5 nearest neighbors
knn_imputer = KNNImputer(n_neighbors=5, weights='distance')
df_imputed = pd.DataFrame(
    knn_imputer.fit_transform(df_numeric),
    columns=numeric_cols,
    index=df.index
)

Iterative Imputation (MICE) models each feature with missing values as a function of other features, iterating until convergence:

from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer
from sklearn.ensemble import RandomForestRegressor

# Basic iterative imputer
iterative_imputer = IterativeImputer(max_iter=10, random_state=42)
df_imputed = pd.DataFrame(
    iterative_imputer.fit_transform(df_numeric),
    columns=numeric_cols,
    index=df.index
)

# Use Random Forest as the estimator for non-linear relationships
rf_imputer = IterativeImputer(
    estimator=RandomForestRegressor(n_estimators=50, random_state=42),
    max_iter=10,
    random_state=42
)
df_imputed_rf = pd.DataFrame(
    rf_imputer.fit_transform(df_numeric),
    columns=numeric_cols,
    index=df.index
)

MICE preserves the relationships between variables better than any other method. The tradeoff is computational cost—it’s significantly slower than simple imputation.

Choosing the Right Strategy

Use this decision framework:

Practical decision process:

1. If missing data is under 5% and appears random, deletion is acceptable
2. For exploratory analysis where speed matters, use mean/median imputation
3. For time series, always try forward fill or interpolation first
4. For predictive modeling where accuracy matters, use KNN or MICE
5. For MNAR data, consider domain-specific approaches or flagging missingness as a feature

# Create a missingness indicator before imputing
# This preserves information about the missing pattern
df['income_was_missing'] = df['income'].isnull().astype(int)
df['income'] = df['income'].fillna(df['income'].median())

Conclusion

Missing data handling isn’t a one-size-fits-all problem. Start by understanding the mechanism—MCAR, MAR, or MNAR—because this determines which methods preserve statistical validity. Visualize missingness patterns before choosing a strategy.

For quick analyses, simple imputation works. For production models where accuracy matters, invest in KNN or iterative imputation. When data is MNAR, no imputation method is truly safe; consider domain expertise or treat missingness as informative.

The worst approach is ignoring missing data or blindly applying dropna(). Every missing value tells a story—your job is to handle it without distorting the narrative your data is trying to tell.