You can spend a week reading Spark's configuration reference and still not speed up your job. Most settings are irrelevant to most workloads. These seven are the ones that actually change how a job performs.
1. spark.sql.shuffle.partitions
The default of 200 is wrong for both small and large data. Too few and each task is huge; too many and you drown in scheduling overhead. Aim for partitions around 128–256 MB each:
spark.conf.set("spark.sql.shuffle.partitions", "600")
2. spark.sql.adaptive.enabled
Adaptive Query Execution re-optimizes at runtime — coalescing partitions, handling skew, switching join strategies. Turn it on and Spark fixes many problems for you:
spark.conf.set("spark.sql.adaptive.enabled", "true")
3. Executor Memory and Cores
Prefer several medium executors over a few giant ones. Around 4–5 cores per executor is the sweet spot — more causes I/O contention:
spark.conf.set("spark.executor.cores", "4") spark.conf.set("spark.executor.memory", "8g")
4. spark.serializer
Kryo serialization is faster and more compact than the Java default. There is almost no reason not to use it:
spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
5. Broadcast Join Threshold
Raise this so Spark broadcasts slightly larger dimension tables and skips the shuffle. Set it just above your largest lookup table:
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "50m")
6. Predicate and Column Pushdown
Read Parquet/ORC and filter early so Spark skips whole row groups. This is mostly about how you write queries — filter and select columns before joins and aggregations, not after.
7. Caching — Only What You Reuse
cache() is powerful and easily misused. Cache a DataFrame you reference multiple times; do not cache something read once, or you pay memory cost for no benefit.
Summary
Get the partition count right, turn on adaptive execution, size executors sanely, use Kryo, tune broadcast joins, push filters down, and cache only reused data. Nail these seven and you have captured the vast majority of the tuning gains available — everything else is fine-print you can ignore until these are solid.