Chunking is a retrieval decision, not preprocessing

Chunking gets treated as a preprocessing step with a size parameter. It is the decision that determines what your system can find at all, and no amount of prompt work recovers from getting it wrong.

The size trade

Small chunks embed precisely and lose context. A paragraph saying "this is not recommended in production" is useless if the sentence establishing what "this" refers to is in the previous chunk.

Large chunks retain context and blur. An embedding averages the meaning of everything in it, so a chunk covering three topics is a mediocre match for all three and a strong match for none.

There is no universal correct size, but there are useful defaults: something in the region of a few hundred tokens with meaningful overlap works for prose, and structured or reference material often wants smaller.

Overlap, and why it is not waste

Overlapping chunks duplicate content, which feels wasteful. It is insurance against the worst failure mode: a fact that sits exactly on a boundary and appears whole in no chunk at all.

With overlap, that fact appears complete in at least one. The storage cost is real and small; the failure it prevents is total for that fact.

Split on structure, not characters

Splitting every N characters cuts mid-sentence and mid-table.

Documents have structure — headings, sections, paragraphs, list items, code blocks — and splitting on those boundaries produces chunks that are self-contained by construction. Then merge small ones and split oversized ones, rather than starting from a character count.

Carrying the heading trail into each chunk helps more than expected: a chunk that knows it came from "Deployment → Rollbacks" is far more retrievable than the same text alone.

Preserve the structure the author already gave you. They chunked the document when they wrote it.

Measure retrieval on its own

This is the part that is nearly always skipped, and it is what turns chunking from taste into engineering.

Take a set of questions with known answers. For each, record whether the chunk containing the answer appears in the retrieved set. That single number — recall at k — tells you whether your chunking works.

Change the strategy, re-measure. Without it you are changing a parameter and evaluating the whole pipeline by eye, where a generation change can mask a retrieval regression.

Metadata is what makes filtering possible

Preserve source, section, page and date through ingestion. Two reasons.

It is what real citations are built from — "according to this document" is not a citation, "page 14, section 3" is.

And it is what lets you filter before searching: only current documents, only this customer's, only this product version. Pre-filtering is extremely common in production and impossible if the metadata was discarded at ingestion time.