Almost every Spark performance question — Why is it slow? Why did it spill? Why is one task stuck? — comes back to partitioning. Once the mental model clicks, the rest of Spark gets much easier to reason about.
A Partition Is a Unit of Work
A Spark DataFrame is split into partitions, and each partition is processed by one task on one core. If your data has 200 partitions, Spark can do 200 chunks of work in parallel — but never more at once than you have cores. Partitions are the atoms of parallelism.
df.rdd.getNumPartitions() # how many chunks your data is split into
Too few partitions and you underuse the cluster; too many and scheduling overhead dominates.
Shuffles Repartition Your Data
Operations like groupBy, join, and distinct require rows with the same key to end up on the same partition. Spark achieves this with a shuffle: it redistributes data across the network by hashing the key. Shuffles are expensive because they move data between executors — minimizing them is much of Spark tuning.
repartition vs coalesce
Two ways to change partition count, and they are not the same:
df.repartition(400) # full shuffle, even distribution, can increase or decrease df.coalesce(50) # no shuffle, merges partitions, decrease only
Use coalesce to shrink partitions cheaply (e.g. before writing fewer output files). Use repartition when you need an even spread or to increase parallelism, accepting the shuffle cost.
Partitioning on Write
When you write data, partitioning by a column physically organizes files into folders, so later reads can skip whole partitions:
df.write.partitionBy("event_date").parquet("events/")
A query filtered on event_date then reads only the relevant folders — this is partition pruning, and it can turn a full scan into a tiny one.
Summary
Partitions are how Spark parallelizes: one partition, one task, one core. Shuffles repartition your data by key and are the main cost to manage. Use repartition and coalesce deliberately, and partition your written data on the columns you filter by. Hold this model and Spark's behavior stops being mysterious.