One line of PySpark, 34% off the runtime

Building a pipeline over roughly 1.9 million records from three different corpora, the most valuable change I made was a single hint — and I nearly did not try it, because the job already ran and the plan looked reasonable.

The setup

Three heterogeneous sources unified into one schema: a large text corpus, a review dataset, and a question-answering set. 1,949,519 records, about 1.04GB, flattened to 3,899,038 rows for analysis.

Then a join: the main dataset against a small lookup table. Five rows.

Why the default plan was wrong

Spark chose a shuffle join, its general-purpose strategy. Both sides get partitioned by the join key and redistributed across the cluster so that matching keys land on the same executor. It is correct, and it works at any scale, which is why it is the default.

It is also enormously wasteful when one side has five rows. Shuffling is expensive: serialise, move across the network, deserialise, sort. Doing that to a table that would fit in a single cache line is pure overhead.

A broadcast join sends the small table to every executor once. Each executor then joins its local partition against a local copy, and the shuffle for that side disappears entirely.

Benchmarked on identical data: 8.996 seconds with the standard join, 5.949 with the broadcast hint. A third of the runtime, for a hint.

Why the optimiser did not do this itself

Spark does broadcast automatically when it knows a side is small — there is a size threshold and an auto-broadcast path. It did not fire here, and the reason is instructive.

The optimiser reasons from statistics. When a side of the join is not a plain table scan but the result of earlier transformations, its size estimate can be wildly off — errors compound through filters and joins, and a conservative estimator will not risk broadcasting something that might be enormous.

So the optimiser was not being stupid. It was being careful with information it did not have. I knew the lookup table was five rows because I built it. Spark only knew it was the output of a chain of operations.

The optimiser is guessing from statistics it may not have. When you know something it does not — like "this table will always be tiny" — say so.

Reading the plan

df.explain() costs nothing and is the single most useful habit in Spark work. It shows the physical plan: the join strategy, where exchanges happen, whether filters were pushed down to the scan.

Most of my Spark wins have come from reading the plan and disagreeing with one line of it, not from adding memory or executors. Specifically, three things are worth looking for. Is the join strategy what you expected? Are filters being pushed down to the source, or is the whole dataset being read and filtered later? And how many exchanges are there, given each is a full shuffle?

The rest of the tuning

Adaptive query execution helps a great deal and is worth having on — it re-plans at runtime with real statistics, which addresses exactly the estimation problem above, including coalescing small partitions and handling skewed joins.

Partition count matters more than people expect. Too few and you cannot use the cluster. Too many and per-task overhead dominates; with small data this is easy to get wrong in the expensive direction, and the default is frequently far too high for a modest job.

Serialisation is worth configuring once. The faster serialiser is not the default for historical reasons and is generally a straight improvement for typical workloads.

What the exercise was really about

The join hint saved three seconds on a benchmark. That is not the point.

The point is that the default was wrong in a way that scales: on a bigger dataset the same unnecessary shuffle costs minutes, not seconds, and it would still have looked normal in the logs. Nothing about the job appeared broken. It ran, it produced correct output, and it wasted a third of its time doing work that did not need doing.

That is the characteristic shape of distributed-systems waste — not errors, just inefficiency that never announces itself. The only way to find it is to read the plan and ask whether each step earns its place.