NumPy - Random Shuffle and Permutation
NumPy provides two primary methods for randomizing array elements: shuffle() and permutation(). The fundamental difference lies in how they handle the original array.
Shuffle
Fisher-Yates Shuffle: Unbiased Random Permutation
Shuffling an array seems trivial. Loop through, swap things around randomly, done. This intuition has led countless developers to write broken shuffle implementations that look correct but produce…
Read more →Apache Spark - Shuffle Operations and Performance
A shuffle occurs when Spark needs to redistribute data across partitions. During a shuffle, Spark writes intermediate data to disk on the source executors, transfers it over the network, and reads it…
Read more →Apache Spark - Avoid Shuffle Operations
A shuffle in Apache Spark is the redistribution of data across partitions and nodes. When Spark needs to reorganize data so that records with the same key end up on the same partition, it triggers a…
Read more →