You launched a Spark job on a 50-node cluster, and 49 tasks finished in seconds while one ran for twenty minutes. That is data skew: a join or aggregation key so lopsided that one partition holds most of the rows. Here is how to diagnose and fix it.
Confirm It Is Skew
Open the Spark UI and look at the stage's task list. Sort by duration. If one or two tasks have vastly higher input size and runtime than the median, you have skew. The giveaway is a single straggler task holding gigabytes while the rest hold megabytes.
Fix 1: Enable Adaptive Query Execution
Modern Spark can split skewed partitions automatically. Turn it on:
spark.conf.set("spark.sql.adaptive.enabled", "true") spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
AQE detects oversized partitions at runtime and splits them into smaller subtasks. For many jobs this alone is enough.
Fix 2: Salt the Skewed Key
When one key dominates a join, spread it across buckets by adding a random salt, then join on the salted key:
from pyspark.sql import functions as F salted = big_df.withColumn("salt", (F.rand() * 10).cast("int")) small_exploded = (small_df .withColumn("salt", F.explode(F.array([F.lit(i) for i in range(10)])))) result = salted.join(small_exploded, ["join_key", "salt"])
The hot key is now spread across ten partitions instead of one.
Fix 3: Broadcast the Small Side
If one side of the join is small enough to fit in memory, broadcast it and skip the shuffle entirely:
from pyspark.sql.functions import broadcast result = big_df.join(broadcast(small_df), "join_key")
No shuffle means no skew, because there is no partitioning by the join key at all.
Summary
Skew is one partition doing everyone's work. First confirm it in the Spark UI, then reach for the cheapest fix that applies: enable adaptive skew handling, broadcast the small side if it fits, and salt the hot key when neither is enough. Balance the partitions and your straggler task disappears — along with most of your runtime.