Processing a million rows without loading a million rows

A script that works on ten thousand records and dies on a million is nearly always doing the same thing: materialising a collection it only needed to walk through.

Each chained method builds another array

In Ruby, map returns an array. Chain select after it and you have two arrays. Chain a third and you have three, each holding the full intermediate result.

For small collections this is irrelevant and the readability is worth it. For a million rows it is the entire problem — peak memory is several multiples of the data, and the garbage collector spends its time on objects that existed only to be passed along.

lazy changes the evaluation model. Elements flow through the whole chain one at a time, and nothing intermediate is retained. The code reads identically; the memory profile does not.

The catch is that a lazy chain needs a terminal call to actually run, and forgetting it produces an enumerator that looks like a result and contains nothing.

The database side is the bigger win

Post.all.each loads every row into memory before the first iteration. On a large table that is the whole table, as objects.

find_each fetches in batches and yields records one at a time, keeping only a batch resident. in_batches gives you the relation per batch, which lets you do set-based work — a bulk update per batch rather than per row.

Batch methods ignore your ordering, because they order by primary key to paginate reliably. Code that depends on a specific order needs a different approach, and quietly getting a different order is a real source of confusion.

Select only what you need

Loading full objects to read two columns is waste on every axis: bytes over the wire, memory, and object allocation.

pluck returns raw values with no model instantiation at all. For a million rows the difference between plucking two columns and instantiating a million objects is not marginal — it is the difference between a script that finishes and one that does not.

Ask for the rows you need, the columns you need, and one batch at a time. Most memory problems are a violation of one of the three.

Streaming out as well as in

The same principle applies to output. Building a CSV of a million rows in memory before sending it has the same shape of problem as loading them.

Rails can stream a response, writing rows as they are produced. Combined with batched reads, an export of any size runs in constant memory — which turns "we cannot export that, it times out" into a solved problem rather than a permanent limitation.