“Slaying the Skewness Monster: A Comical Guide to Data Processing in Production”

My articles are open to everyone; non-member readers can read the full article by clicking this **link**.

For all Spark Data engineers out there, I’m sure you can relate to this scenario:

Lets start with a tale of the dreaded data skew, a sneaky little bugger that likes to hide in the shadows during development but then pounces on you like a hungry lion when you start processing real-world data. It’s like a ninja that sneaks up on you when you least expect it — you think you’re in the clear and then BAM! Your code starts acting like a diva with a serious case of the Mondays.

Skewed data Spark UI

Imagine this : You just emerged from your coding cave, hair disheveled and eyes bloodshot from hours of typing away furiously. But wait! Your code is actually working! It’s handling terabytes of data like a boss and all your tests are green!

You feel like a coding superhero, ready to take on any challenge that comes your way (except maybe a crowded coffee shop, that’s just too much for any hero to handle).

So, you’re beaming with pride after your coding triumph and you decide to brag about it to a colleague. And then they have the nerve to suggest you put your code to the test with some production data. “Ha! Who needs it?” you scoff.

“My code is perfect, why mess with perfection?” It’s like they’re suggesting you take your new Lamborghini for a spin on a dirt road — it’s just silly.

With some extra time on your hands, you decide to take up your co-worker’s challenge. You copy over some production data and hit the run button, thinking this will be a piece of cake. But then, one minute turns into an hour, and an hour turns into two… and suddenly all hell breaks loose. Your code starts acting like a grumpy toddler who missed nap time — throwing tantrums left and right, crashing at the slightest challenge, and generally making your life a living nightmare.

It’s like a scene straight out of a horror movie, except instead of a monster chasing you, it’s just lines of code.

Some of you out there might think this story sounds like a nightmare, while others might just nod their heads and say “yup, been there, done that.”

Let’s work together to understand how can we take care of this devil in production :

What is Skewness ?

Skewness is like the Goldilocks of statistics — it measures whether the distribution of values is too much to the left, too much to the right, or just right around the mean value.

If it’s too much to the left, we call it negative skewness.

if it’s too much to the right, we call it positive skewness.

if it’s just right, with values evenly spread around the mean like a cozy blanket, we call it zero skewness.

if it’s a constant, well, we call it undefined skewness — which is a fancy way of saying “it is what it is.”

Spark provides the SQL built-in function skewness that calculates the skewness of a numeric Dataframe column. In the Spark code base you will find the applied formula.

In this article, I will cover my learnings(in a humorous way) as how did I handled data skewness with my experience along with standard approaches to handle data skew.

Skewness can be good as It can help you identify outliers and better understand the distribution of your data.

Skewness is like when you’re at a party and you notice that one person is dancing wildly while everyone else is standing around awkwardly — that person is the outlier, and they’re skewing the distribution of dance moves.

Skewness can be bad as It can make it harder to analyze your data and draw meaningful insights.

Skewness is like when you’re playing Jenga and one person keeps removing all the blocks from the same side — the tower becomes so skewed that it eventually collapses.

Impact of data skewness on performance :

Lets recall how spark job works :

• When an action(save, count) is called , A spark job is created and computation of all the transformations on the way is triggered.
• Each job breaks down into one or more stages. The number of stages depend on the amount of shuffles that are required to perform the action and two stages are separated by a shuffle.
• A stage represent a set of tasks that are executed together to compute the same operation on multiple executors.

When your data is skewed it means it is unevenly distributed across the partitions. Because a partition is the smallest data unit available in Spark, the task duration for processing that skewed data is also unbalanced. Overall,

• a stage is only as fast as its slowest task, and
• a subsequent stage can not start until the previous stage has finished.

This means you could end up having many tasks being done with their workload quickly but one task might take a lot of time. This slow task will delay the entire stage and consequently the entire application.

Following are the ways to handle Data skewness :

1. Handle separately all the skewed data but this might fail with dynamic data on production as this deals with filtering out the skewed values and then take union with the result post processing.This is a hack and may fail at any time.
2. Broadcast Hash Join: Spark can broadcast the smaller DataFrame to all the worker nodes and perform the join locally, avoiding the shuffling of data and the consequent skewness. This is based on the assumption that one of the dataframe being used in joining is smaller than 8GB which may not be true for all problem statements.
3. Key Salting: Salting is a technique used to prevent data skewness when performing joins in Spark DataFrames. It involves adding a random value (known as a “salt”) to a join key column to distribute data more evenly across partitions.

Imagine you’re hosting a dinner party and trying to assign guests to different tables based on their food preferences. If you have a lot of guests who love pizza, you could add a “salt” to their names (like “PizzaLover_1”, “PizzaLover_2”, etc.) to prevent them from all being assigned to the same table.

In Spark, you can implement salting using the “rand()” function to generate a random value to add to the join key column. Here’s an example:

from pyspark.sql.functions import rand
# create DataFrame with join key column
df1 = spark.createDataFrame([(1, "A"), (2, "B"), (3, "C")], ["id", "value"])
df2 = spark.createDataFrame([(1, "X"), (2, "Y"), (3, "Z")], ["id", "value"])
# add salt to join key column
salted_df1 = df1.withColumn("salt", rand())
salted_df2 = df2.withColumn("salt", rand())
# join on salted join key column
joined_df = salted_df1.join(salted_df2, ["id", "salt"])

While salting can help improve performance and prevent data skewness in joins, it can also lead to increased memory usage and more shuffling of data.

Additionally, if the number of partitions is too small or too large, salting may not be as effective. So, just like adding too much salt to a dish, too much salting in Spark can ruin the flavor of your data processing!

Bonus:

AQE[Adaptive Query Execution] from spark 3 onwards definitiely helping for average scenarios but not helpful for larger datasets processing and may introduce additional skewness.It doesnt provide guranteed skewness removal.

Skewness removal when using window function is difficult and you need custom solution around salting to handle it .Details cak be found here .

Conclusion:

In summary, while data skews may not be the Godzilla of problems, it’s always wise to nip them in the bud by anticipating their occurrence and testing our codes on skewed datasets. After all, it’s better to be safe than sorry, or in this case, better to be skewed than screwed!

All my spark based blogs are here, it will be helpful for you :

1. Sparking Up the Interview Room 1: Tackling Spark Interview Scenario-based Questions with Wit and Wisdom!
2. Sparking Up the Interview Room 3: Tackling Spark Interview Scenario-based Questions with Wit and Wisdom!
3. Sparking Up the Interview Room 2: Tackling Spark Interview Scenario-based Questions with Wit and Wisdom!
4. Spark Interview Questions
5. “Get Grouped and Groovy: How Spark Grouping Sets Can Turn Your Data Analysis into a Party-Advance”
6. How Compression Codecs Save Spark from Suffocating on Data!
7. Spark Shuffle Unveiled: How to Turn Your Cluster into a Fire-Breathing Beast!”
8. Spark Series: Partition Discovery & Production Learning

References:

Images :

[embed]Dancer Silhouette Images — Free Download on Freepik When dancing, the number of different poses we adopt is huge. Imagine if we captured many of these poses and turned…www.freepik.com

https://www.istockphoto.com/photo/taking-one-block-from-wooden-blocks-tower-gm547208858-98853379

[embed]Data Skew in Apache Spark Understanding Why Data Skews Occur and How to Deal with Themselectfrom.dev

https://michaelheil.medium.com/understanding-common-performance-issues-in-apache-spark-deep-dive-data-skew-e962909f3d07