Spark Scala - RDD Operations
Resilient Distributed Datasets (RDDs) are Spark’s original abstraction for distributed data processing. While DataFrames and Datasets have become the preferred API for most workloads, understanding…
Read more →Rdd
PySpark - RDD Partitioning (getNumPartitions, repartition)
• RDD partitioning directly impacts parallelism and performance—understanding getNumPartitions() helps diagnose processing bottlenecks and optimize cluster resource utilization
PySpark - RDD Persistence (cache, persist)
• RDD persistence stores intermediate results in memory or disk to avoid recomputation, critical for iterative algorithms and interactive analysis where the same dataset is accessed multiple times
Read more →PySpark - RDD reduceByKey with Examples
from pyspark.sql import SparkSession
Read more →PySpark - RDD sortByKey with Examples
The sortByKey() transformation operates exclusively on pair RDDs—RDDs containing key-value tuples. It sorts the RDD by keys and returns a new RDD with elements ordered accordingly. This operation…
PySpark - RDD Transformations (map, filter, flatMap)
• RDD transformations are lazy operations that define a computation DAG without immediate execution, enabling Spark to optimize the entire pipeline before materializing results
Read more →PySpark - RDD vs DataFrame - When to Use Which
• RDDs provide low-level control and are essential for unstructured data or custom partitioning logic, but lack automatic optimization and require manual schema management
Read more →PySpark RDD Tutorial - Complete Guide with Examples
RDDs are the fundamental data structure in Apache Spark. They represent an immutable, distributed collection of objects that can be processed in parallel across a cluster. While DataFrames and…
Read more →PySpark: RDD vs DataFrame Guide
PySpark gives you two primary ways to work with distributed data: RDDs and DataFrames. This isn’t redundant design—it reflects a fundamental trade-off between control and optimization.
Read more →PySpark - RDD Actions (collect, count, first, take)
PySpark operations fall into two categories: transformations and actions. Transformations are lazy—they build a DAG (Directed Acyclic Graph) of operations without executing anything. Actions trigger…
Read more →PySpark - RDD Broadcast Variables
Broadcast variables provide an efficient mechanism for sharing read-only data across all nodes in a Spark cluster. Without broadcasting, Spark serializes and sends data with each task, creating…
Read more →PySpark - RDD groupByKey with Examples
• groupByKey() creates an RDD of (K, Iterable[V]) pairs by grouping values with the same key, but should be avoided when reduceByKey() or aggregateByKey() can accomplish the same task due to…
PySpark - RDD join Operations
• RDD joins in PySpark support multiple join types (inner, outer, left outer, right outer) through operations on PairRDDs, where data must be structured as key-value tuples before joining
Read more →PySpark - Pair RDD Operations
• Pair RDDs are the foundation for distributed key-value operations in PySpark, enabling efficient aggregations, joins, and grouping across partitions through hash-based data distribution.
Read more →PySpark - Create RDD from List (parallelize)
Resilient Distributed Datasets (RDDs) are the fundamental data structure in PySpark, representing immutable, distributed collections that can be processed in parallel across cluster nodes. While…
Read more →PySpark - Create RDD from Text File
Resilient Distributed Datasets (RDDs) represent PySpark’s fundamental abstraction for distributed data processing. While DataFrames have become the preferred API for structured data, RDDs remain…
Read more →PySpark - Convert RDD to DataFrame
RDDs (Resilient Distributed Datasets) represent Spark’s low-level API, offering fine-grained control over distributed data. DataFrames build on RDDs while adding schema information and query…
Read more →PySpark - Create DataFrame from RDD
• DataFrames provide significant performance advantages over RDDs through Catalyst optimizer and Tungsten execution engine, making conversion worthwhile for complex transformations and SQL operations.
Read more →Apache Spark - RDD vs DataFrame vs Dataset
Resilient Distributed Datasets (RDDs) are Spark’s fundamental data structure—immutable, distributed collections of objects partitioned across a cluster. They expose low-level transformations and…
Read more →