Solving Spark Small Files After Shuffle with Adaptive Coalesce

I’ve run into the small files problem in Spark more times than I can count — especially after shuffle stages where you end up with thousands of tiny Parquet files that kill downstream performance. It’s frustrating because the logic of your job is fine, but the physical layout hurts. In my environment, enabling adaptive query execution (AQE) and specifically turning on coalesce post-shuffle has made a repeatable difference. Here’s how I approach it.

Why small files happen after shuffle

When Spark performs a shuffle — say, during a groupBy or join — it writes intermediate data to disk partitioned by the shuffle keys. If the data is skewed or the initial partitioning is too fine, you get lots of small output files. These aren’t just annoying; they cause excessive task launches, metadata overhead, and poor columnar scan efficiency in formats like Parquet or ORC. I’ve seen jobs where 90% of the runtime was just task scheduling overhead from 50,000+ files under 10KB each.

How adaptive coalesce helps

Spark 3.0+ introduced adaptive query execution, which can replan stages based on runtime statistics. One part of this is spark.sql.adaptive.coalescePartitions.enabled. When true, after a shuffle stage, Spark evaluates whether the resulting partitions are too small and merges them into fewer, larger ones before proceeding. This doesn’t change the logic — it just reduces the number of write tasks and output files.

I set this alongside the core AQE flag:

spark.sql.adaptive.enabled=true
spark.sql.adaptive.coalescePartitions.enabled=true
spark.sql.adaptive.coalescePartitions.minPartitionNum=10
spark.sql.adaptive.advisoryPartitionSizeInBytes=128MB

The advisory size tells Spark what a good partition size looks like; the minPartitionNum prevents over-coalescing. In my ETL pipelines on YARN, this dropped post-shuffle file counts from ~20k to under 500 with no data loss and a 3–5x speedup in the write phase.

Verifying it’s working

You can check if coalesce happened in the Spark UI. Look for the shuffle read stage — if the number of output partitions is less than the number of input partitions to that stage, and you see a "Coalesce Partitions" node in the physical plan, it’s active. I also like to glance at the SQL tab for the explain plan:

EXPLAIN COST SELECT user_id, COUNT(*) FROM events GROUP BY user_id

If AQE is on, you’ll see a note about adaptive planning and possibly a coalesce step.

Caveats I’ve hit

This isn’t magic. If your data is extremely skewed, coalescing won’t fix the underlying imbalance — you might still get a few huge partitions and many small ones. In those cases, I combine this with salting or custom partitioning. Also, coalescing adds a small barrier shuffle step, so if your shuffle is already optimal, you might add latency. That’s why I tune the advisory size based on actual cluster I/O and file format.

As I mentioned before in my tuning notes, I always start with the defaults and adjust upward only if I see persistent small files after enabling AQE.

When to use it

I enable this by default on all production Spark jobs that involve shuffles — especially ETL, aggregations, and joins writing to data lakes. It’s low-risk, reversible, and the gains are consistent. If you’re using Spark 3.2+ and seeing small files after shuffle, turn it on and measure the output.

Final thoughts

The small files problem isn’t going away, but adaptive coalesce gives you a runtime lever to control it without rewriting jobs. Pair it with proper partitioning and monitoring, and you’ll spend less time fighting file counts and more time on actual data work.

Cover image: learn_tek · CC0 (Openverse / kamu malı) · https://www.flickr.com/photos/153724200@N07/27774351928