Spark Scala - Cache and Persist
Spark’s lazy evaluation model means transformations build up a lineage graph that gets executed only when you call an action. This is elegant for optimization, but it has a cost: every action…
Read more →Cache
PySpark - RDD Persistence (cache, persist)
• RDD persistence stores intermediate results in memory or disk to avoid recomputation, critical for iterative algorithms and interactive analysis where the same dataset is accessed multiple times
Read more →PySpark - Cache and Persist DataFrame
PySpark operates on lazy evaluation, meaning transformations like filter(), select(), and join() aren’t executed immediately. Instead, Spark builds a logical execution plan and only computes…
MySQL Query Cache: Configuration and Limitations
• MySQL Query Cache was deprecated in MySQL 5.7.20 and removed entirely in MySQL 8.0 due to scalability issues and lock contention in multi-core environments
Read more →LRU Cache: Least Recently Used Implementation
Caching is the art of keeping frequently accessed data close at hand. But caches have limited capacity, so when they fill up, something has to go. The eviction policy—the rule for deciding what gets…
Read more →LFU Cache: Least Frequently Used Implementation
Least Frequently Used (LFU) caching takes a fundamentally different approach than its more popular cousin, LRU. While LRU evicts the item that hasn’t been accessed for the longest time, LFU evicts…
Read more →How to Cache a DataFrame in PySpark
If you’ve ever watched a Spark job run the same expensive transformation multiple times, you’ve experienced the cost of ignoring caching. Spark’s lazy evaluation model means it doesn’t store…
Read more →Design a Distributed Cache: Memcached/Redis Architecture
Every high-scale system eventually hits the same wall: database latency becomes the bottleneck. Your PostgreSQL instance handles 10,000 queries per second beautifully, but at 50,000 QPS, response…
Read more →Apache Spark - When to Cache vs Persist vs Checkpoint
Spark’s lazy evaluation is both its greatest strength and a subtle performance trap. When you chain transformations, Spark builds a Directed Acyclic Graph (DAG) representing the lineage of your data….
Read more →