Your RAG demo is lying to you

I built an interactive course on retrieval-augmented generation for this site, and the most useful thing I learned building it had nothing to do with retrieval quality. It was about how easy it is to draw a confident conclusion from a system that isn't the one you think you're measuring.

The lesson that almost shipped

One lesson teaches hybrid search: combining keyword ranking with vector ranking because embeddings are weak at exact terms. I needed a question where vector search genuinely misses and keyword search hits. I found one quickly — "import fees". The vector search returned the membership page, which says "no fee"; keyword search found the shipping page that says Europe orders "may be charged import fees". A perfect example.

Except it wasn't. The course embeds the documents with a real sentence-embedding model, but questions a learner types have to be embedded in the browser, so they use a cheaper approximation: an IDF-weighted average of the model's vectors for each word. Before writing the lesson I ran "import fees" through the full model. It ranked the shipping page first. The miss I was about to teach was an artefact of my approximation, not a property of embeddings.

I threw it away and searched for a case the real model gets wrong. "one hour", at 60-word chunks, is one: the full model returns a shipping chunk about tracking, while the password page literally says the reset link "works for one hour". That lesson is true, so that's the lesson.

Every cheap layer is a different system

This pattern is everywhere in RAG work, and it is rarely this visible. Each layer you add to make retrieval cheaper changes behaviour in its own way:

Dimensionality reduction. I reduced vectors to 128 dimensions with PCA to keep the payload small. That kept about three-quarters of the variance. Rankings mostly survive; close calls don't always.

Quantisation. Storing vectors as 8-bit integers is usually harmless for ranking, but "usually" is a claim you test, not assume.

Smaller or different models for development than for production. Your offline eval measures the model you evaluated with.

Approximate nearest-neighbour indexes. Fast, and occasionally they skip the true nearest neighbour. Fine for most traffic — not fine as the basis of a conclusion.

None of these are mistakes. All of them mean your demo, your eval and your production system can disagree, and the disagreement shows up exactly where you are looking for interesting behaviour: the edge cases.

What I do now

Keep a small set of questions whose correct chunks you know, and run them through every configuration you care about — the full model, the reduced one, the index you actually deploy. When they disagree, that disagreement is the finding.

Separate "what the model does" from "what my pipeline does" when you explain a failure. "Embeddings can't handle exact phrases" and "my averaged word vectors can't" are different claims with different fixes.

And when you teach something, verify the example against the real thing. A lesson built on an artefact doesn't just mislead one reader; it hands them an intuition they will apply confidently for years.

The most dangerous evaluation result is the one that confirms what you already believed.