R tidyr - separate() Column into Multiple
• The separate() function splits one column into multiple columns based on a delimiter, with automatic type conversion and flexible handling of edge cases through parameters like extra and fill
Column
PySpark - Trim/Ltrim/Rtrim Whitespace from Column
Whitespace in data columns is a silent killer of data quality. You’ve probably encountered it: joins that mysteriously fail to match, duplicate records after grouping, or inconsistent filtering…
Read more →PySpark - Update Column Value Conditionally
Conditional column updates are fundamental operations in PySpark, appearing in virtually every data pipeline. Whether you’re cleaning messy data, engineering features for machine learning models, or…
Read more →PySpark - Substring from Column
String manipulation is fundamental to data engineering workflows, especially when dealing with raw data that requires cleaning, parsing, or transformation. PySpark’s DataFrame API provides a…
Read more →PySpark - Split String Column into Multiple Columns
Working with delimited string data is one of those unglamorous but essential tasks in data engineering. You’ll encounter it constantly: CSV-like data embedded in a single column, concatenated values…
Read more →PySpark - Replace Column Values (regexp_replace)
Data cleaning is messy. Real-world datasets arrive with inconsistent formatting, unwanted characters, and patterns that vary just enough to make simple string replacement useless. PySpark’s…
Read more →PySpark - Rename Column Name in DataFrame
PySpark DataFrames are the backbone of distributed data processing, but real-world datasets rarely arrive with clean, consistent column names. You’ll encounter spaces, special characters,…
Read more →PySpark - Map Column Values Using when/otherwise
When working with large-scale data in PySpark, you’ll frequently need to transform column values based on conditional logic. Whether you’re categorizing continuous variables, cleaning data…
Read more →PySpark - Length of String Column
Calculating string lengths is a fundamental operation in data engineering workflows. Whether you’re validating data quality, detecting truncated records, enforcing business rules, or preparing data…
Read more →PySpark - Get Column Names as List
Working with PySpark DataFrames frequently requires programmatic access to column names. Whether you’re building dynamic ETL pipelines, validating schemas across environments, or implementing…
Read more →PySpark - Filter Rows by Column Value
Filtering rows is one of the most fundamental operations in any data processing workflow. In PySpark, you’ll spend a significant portion of your time selecting subsets of data based on specific…
Read more →PySpark - Distinct Values in Column
Finding distinct values in PySpark columns is a fundamental operation in big data processing. Whether you’re profiling a new dataset, validating data quality, removing duplicates, or analyzing…
Read more →PySpark - Drop Column from DataFrame
Column removal is one of the most frequent operations in PySpark data pipelines. Whether you’re cleaning raw data, reducing memory footprint before expensive operations, removing personally…
Read more →PySpark - Explode Array Column to Rows
PySpark DataFrames frequently contain array columns when working with semi-structured data sources like JSON, Parquet files with nested schemas, or aggregated datasets. While arrays are efficient for…
Read more →PySpark - Cast Column to Different Type
Type casting in PySpark is a fundamental operation you’ll perform constantly when working with DataFrames. Unlike pandas where type inference is aggressive, PySpark often reads data with conservative…
Read more →PySpark - Convert Column to List (collect)
One of the most common operations when working with PySpark is extracting column data from a distributed DataFrame into a local Python list. While PySpark excels at processing massive datasets across…
Read more →PySpark - Add Column with Constant/Literal Value
• Use lit() from pyspark.sql.functions to add constant values to PySpark DataFrames—it handles type conversion automatically and works seamlessly with the Catalyst optimizer
PySpark - Add New Column to DataFrame (withColumn)
The withColumn() method is the workhorse of PySpark DataFrame transformations. Whether you’re deriving new features, applying business logic, or cleaning data, you’ll use this method constantly. It…
PySpark - Apply Function to Column (withColumn + UDF)
PySpark DataFrames are immutable, meaning you can’t modify columns in place. Instead, you create new DataFrames with transformed columns using withColumn(). The decision between built-in functions…
PySpark - Add Auto-Increment Column to DataFrame
PySpark DataFrames don’t have a native auto-increment column like traditional SQL databases. This becomes problematic when you need unique row identifiers for tracking, joining datasets, or…
Read more →Pandas - Sort DataFrame by Column (sort_values)
• The sort_values() method is the primary way to sort DataFrames by one or multiple columns, replacing the deprecated sort() and sort_index() methods for column-based sorting
Pandas - Select Single Column from DataFrame
The most common approach to selecting a single column uses bracket notation with the column name as a string. This returns a Series object containing the column’s data.
Read more →Pandas - Set/Reset Column as Index
• Setting a column as an index transforms it from regular data into row labels, enabling faster lookups and more intuitive data alignment—use set_index() for single or multi-level indexes without…
Pandas - Sort by Column Data Type (Custom Sort)
• Pandas doesn’t natively sort by column data types, but you can create custom sort keys using dtype information to reorder columns programmatically
Read more →Pandas - Replace Values in Column
The replace() method is the most versatile approach for substituting values in a DataFrame column. It works with scalar values, lists, and dictionaries.
Pandas - Rename Column by Index
When working with DataFrames from external sources, you’ll frequently encounter datasets with auto-generated column names, duplicate headers, or names that don’t follow Python naming conventions….
Read more →Pandas - Rename Column Names
The rename() method is the most versatile approach for changing column names in Pandas. It accepts a dictionary mapping old names to new names and returns a new DataFrame by default.
Pandas - Rank Values in Column
• Pandas provides multiple ranking methods (average, min, max, first, dense) that handle tied values differently, with the rank() method offering fine-grained control over ranking behavior
Pandas - Merge with Indicator Column
The indicator parameter in pd.merge() adds a special column to your merged DataFrame that tracks where each row originated. This column contains one of three categorical values: left_only,…
Pandas - Move Column to First/Last Position
The most efficient way to move a column to the first position is combining insert() and pop(). The pop() method removes and returns the column, while insert() places it at the specified index.
Pandas - Map Values in Column Using Dictionary
The map() method transforms values in a pandas Series using a dictionary as a lookup table. This is the most efficient approach for replacing categorical values.
Pandas - Insert Column at Specific Position
• Pandas provides multiple methods to insert columns at specific positions: insert() for in-place insertion, assign() with column reordering, and direct dictionary manipulation with…
Pandas - GroupBy Single Column
The groupby() method partitions a DataFrame based on unique values in a specified column. This operation doesn’t immediately compute results—it creates a GroupBy object that holds instructions for…
Pandas - Get Column Names as List
• Pandas DataFrames provide multiple methods to extract column names, with df.columns.tolist() being the most explicit and list(df.columns) offering a Pythonic alternative
Pandas - Explode List Column to Rows
• The explode() method transforms list-like elements in a DataFrame column into separate rows, maintaining alignment with other columns through automatic index duplication
Pandas - Format Datetime Column (strftime)
• The strftime() method converts datetime objects to formatted strings using format codes like %Y-%m-%d, while dt.strftime() applies this to entire DataFrame columns efficiently
Pandas - Drop Column from DataFrame
• Pandas offers multiple methods to drop columns: drop(), pop(), direct deletion with del, and column selection—each suited for different use cases and performance requirements
Pandas - Create DataFrame with Column Names
• DataFrames can be created from dictionaries, lists, or NumPy arrays with explicit column naming using the columns parameter or dictionary keys
Pandas - Convert Column to String
• Use astype(str) for simple conversions, map(str) for element-wise control, and apply(str) when integrating with complex operations—each method handles null values differently
Pandas - Change Column Data Type (astype)
• The astype() method is the primary way to convert DataFrame column types in pandas, supporting conversions between numeric, string, categorical, and datetime types with explicit control over the…
Pandas - Convert Column to Categorical
Categorical data represents a fixed set of possible values, typically strings or integers representing discrete groups. In Pandas, the categorical dtype stores data internally as integer codes mapped…
Read more →Pandas - Convert Column to Datetime
The pd.to_datetime() function converts string or numeric columns to datetime objects. For standard ISO 8601 formats, Pandas automatically detects the pattern:
Pandas - Convert Column to Float
The astype() method provides the most straightforward approach for converting a pandas column to float when your data is already numeric or cleanly formatted.
Pandas - Convert Column to Integer
• Converting columns to integers in Pandas requires handling null values first, as standard int types cannot represent missing data—use Int64 (nullable integer) or fill/drop nulls before conversion
Read more →Pandas - Apply Function to Column
• The apply() method transforms DataFrame columns using custom functions, lambda expressions, or built-in functions, offering more flexibility than vectorized operations for complex transformations
Pandas - Add Column Based on Another Column
The simplest way to add a column based on another is through direct arithmetic operations. Pandas broadcasts these operations across the entire column efficiently.
Read more →Pandas - Add Column with Default/Constant Value
• Adding constant columns in Pandas can be done through direct assignment, assign(), or insert() methods, each with specific use cases for performance and readability
Pandas - Add New Column to DataFrame
The simplest method to add a column is direct assignment using bracket notation. This approach works for scalar values, lists, arrays, or Series objects.
Read more →How to Find the Column Space of a Matrix in Python
• The column space of a matrix represents all possible linear combinations of its column vectors and reveals the true dimensionality of your data, making it essential for feature selection and…
Read more →How to Filter by Column Value in Pandas
Filtering DataFrames by column values is something you’ll do constantly in pandas. Whether you’re cleaning data, preparing features for machine learning, or generating reports, selecting rows that…
Read more →How to Explode a Column in Pandas
When working with real-world data, you’ll frequently encounter columns containing list-like values. Maybe you’re parsing JSON from an API, dealing with multi-select form fields, or processing…
Read more →How to Explode a Column in Polars
Data rarely arrives in the clean, normalized format you need. JSON APIs return nested arrays. Aggregation operations produce list columns. CSV files contain comma-separated values stuffed into single…
Read more →How to Delete a Column in Pandas
Deleting columns from a DataFrame is one of the most frequent operations in data cleaning. Whether you’re removing irrelevant features before model training, dropping columns with too many null…
Read more →How to Delete a Column in Polars
Deleting columns from a DataFrame is one of the most common data manipulation tasks. Whether you’re cleaning up temporary calculations, removing sensitive data before export, or trimming down a wide…
Read more →How to Delete a Column in PySpark
Column deletion is one of those operations you’ll perform constantly in PySpark. Whether you’re cleaning up raw data, removing sensitive fields before export, trimming unnecessary columns to reduce…
Read more →How to Convert Column to Datetime in Pandas
Every data analysis project involving dates starts the same way: you load a CSV, check your dtypes, and discover your date column is stored as object (strings). This is the default behavior, and…
How to Apply a Function to a Column in Pandas
Applying functions to columns is one of the most common operations in pandas. Whether you’re cleaning messy text data, engineering features for a machine learning model, or transforming values based…
Read more →How to Add a New Column in Pandas
Adding columns to a Pandas DataFrame is one of the most common operations you’ll perform in data analysis. Whether you’re calculating derived metrics, categorizing data, or preparing features for…
Read more →How to Add a New Column in Polars
If you’re coming from pandas, your first instinct might be to write df['new_col'] = value. That won’t work in Polars. The library takes an immutable approach to DataFrames—every transformation…
How to Add a New Column in PySpark
Adding columns to a PySpark DataFrame is one of the most common transformations you’ll perform. Whether you’re calculating derived metrics, categorizing data, or preparing features for machine…
Read more →Apache Spark - Column Pruning
Column pruning is one of Spark’s most impactful automatic optimizations, yet many developers never think about it—until their jobs run ten times slower than expected. The concept is straightforward:…
Read more →