Executor

Jan 06, 2025 Engineering

Apache Spark - Executor Memory and Cores Configuration

Apache Spark’s performance lives or dies by how you configure executor memory and cores. Get it wrong, and you’ll watch jobs crawl through excessive garbage collection, crash with cryptic…

Jan 05, 2025 Data Engineering

Apache Spark - Driver and Executor Explained

Apache Spark uses a master-slave architecture where the driver program acts as the master and executors function as workers. The driver runs your main() function, creates the SparkContext, and…