Spark Scala - Read CSV File
CSV files refuse to die. Despite the rise of Parquet, ORC, and Avro, you’ll still encounter CSV in nearly every data engineering project. Legacy systems export it. Business users create it in Excel….
Read more →Csv
Scala - Read CSV File
For simple CSV files without complex quoting or escaping, Scala’s standard library provides sufficient functionality. Use scala.io.Source to read files line by line and split on delimiters.
R - Write CSV File (write.csv / readr::write_csv)
The write.csv() function is R’s built-in solution for exporting data frames to CSV format. It’s a wrapper around write.table() with sensible defaults for comma-separated values.
R - Read CSV File (read.csv / readr::read_csv)
• R offers multiple CSV reading methods—base R’s read.csv() provides universal compatibility while readr::read_csv() delivers 10x faster performance with better type inference
Python - Read/Write CSV Files
The csv module provides straightforward methods for reading CSV files. The csv.reader() function returns an iterator that yields each row as a list of strings.
PySpark - Write DataFrame to CSV File
Writing a DataFrame to CSV in PySpark is straightforward using the DataFrameWriter API. The basic syntax uses the write property followed by format specification and save path.
PySpark - Read Multiple CSV Files
The simplest approach to reading multiple CSV files uses wildcard patterns. PySpark’s spark.read.csv() method accepts glob patterns to match multiple files simultaneously.
PySpark - Read CSV File into DataFrame
PySpark’s spark.read.csv() method provides the simplest approach to load CSV files into DataFrames. The method accepts file paths from local filesystems, HDFS, S3, or other distributed storage…
PySpark - Read CSV with Custom Schema
• Defining custom schemas in PySpark eliminates costly schema inference and prevents data type mismatches that cause runtime failures in production pipelines
Read more →PySpark - Read CSV with Header and InferSchema
• PySpark’s inferSchema option automatically detects column data types by sampling data, but adds overhead by requiring an extra pass through the dataset—use it for exploration, disable it for…
PySpark - Convert DataFrame to CSV
PySpark DataFrames are the backbone of distributed data processing, but eventually you need to export results for reporting, data sharing, or integration with systems that expect CSV format. Unlike…
Read more →Pandas - Write DataFrame to CSV (to_csv)
• The to_csv() method provides extensive control over CSV output including delimiters, encoding, column selection, and header customization with 30+ parameters for precise formatting
Pandas - Read CSV File (read_csv)
The read_csv() function reads comma-separated value files into DataFrame objects. The simplest invocation requires only a file path:
Pandas - Read CSV Skip Rows/Header
• Use skiprows parameter with integers, lists, or callable functions to exclude specific rows when reading CSV files, reducing memory usage and processing time for large datasets
Pandas - Read CSV with Custom Delimiter
The read_csv() function in Pandas defaults to comma separation, but real-world data files frequently use alternative delimiters. The sep parameter (or its alias delimiter) accepts any string or…
Pandas - Read CSV with Different Encodings
• CSV files can have various encodings (UTF-8, Latin-1, Windows-1252) that cause UnicodeDecodeError if not handled correctly—detecting and specifying the right encoding is critical for data integrity
Read more →NumPy - Read CSV with np.genfromtxt()
While pandas dominates CSV loading in data science workflows, np.genfromtxt() offers advantages when you need direct NumPy array output without pandas overhead. For numerical computing pipelines,…
How to Write to CSV in Pandas
Every data pipeline eventually needs to export data somewhere. CSV remains the universal interchange format—it’s human-readable, works with Excel, imports into databases, and every programming…
Read more →How to Write to CSV in Polars
Polars has rapidly become the go-to DataFrame library for Python developers who need speed. Built in Rust with a lazy evaluation engine, it consistently outperforms pandas by 10-100x on common…
Read more →How to Write to CSV in PySpark
CSV remains the lingua franca of data exchange. Despite its limitations—no schema enforcement, no compression by default, verbose storage—it’s universally readable. When you’re processing terabytes…
Read more →How to Read CSV Files in Pandas
CSV files remain the lingua franca of data exchange. Despite the rise of Parquet, JSON, and database connections, you’ll encounter CSVs constantly—from client exports to API downloads to legacy…
Read more →How to Read CSV Files in Polars
Polars has rapidly become the go-to DataFrame library for Python developers who need speed without sacrificing usability. Built in Rust with a Python API, it consistently outperforms pandas on CSV…
Read more →How to Read CSV Files in PySpark
CSV files refuse to die. Despite better alternatives like Parquet, Avro, and ORC, you’ll encounter CSV data constantly in real-world data engineering. Vendors export it, analysts create it, legacy…
Read more →