Your Spark job runs, but it is inexplicably slow, and the logs are full of lines like Spilling in-memory map of 512 MB to disk. That is the signal that Spark ran out of execution memory and started writing intermediate data to disk — trading fast RAM for slow I/O.
What Spill Actually Is
During a shuffle, sort, or aggregation, Spark buffers data in execution memory. When that buffer fills, it writes ("spills") the overflow to local disk to avoid an out-of-memory crash. Spill is a safety valve, not a failure — but every spilled byte gets written and re-read, so heavy spill wrecks performance.
Why It Happens
Three common causes:
- ▸Partitions too large. Fewer, bigger partitions mean more data per task and more pressure on execution memory.
- ▸Wide aggregations. A
groupBywith many keys or a large sort holds a lot in memory at once. - ▸Skew. One fat partition spills while the rest do not.
Fix: Increase Shuffle Partitions
The default of 200 shuffle partitions is often too few for large data. More partitions means less data per task:
spark.conf.set("spark.sql.shuffle.partitions", "800")
Aim for partitions in the low hundreds of MB each. This is the single most effective knob for spill.
Fix: Give Executors More Memory (Carefully)
Raising executor memory helps, but only up to a point — huge heaps cause long GC pauses:
spark.conf.set("spark.executor.memory", "8g") spark.conf.set("spark.memory.fraction", "0.7") # more room for execution+storage
Prefer more, smaller executors over a few giant ones.
Fix: Reduce What You Shuffle
Filter and project before the shuffle so less data moves. Push predicates down, drop unused columns early, and pre-aggregate when you can. The cheapest byte to keep in memory is the one you never shuffled.
Summary
Disk spill means execution memory overflowed mid-shuffle. Confirm it in the logs, then raise spark.sql.shuffle.partitions so each task handles less, give executors headroom without creating GC monsters, and shrink the data before it shuffles. Keep the shuffle in RAM and the job speeds back up.