Your RAG chatbot doesn't have a model problem

The most common request I get is a version of: our chatbot hallucinates, can you swap in a better model?

Almost always, the model is the wrong place to look. Not because models do not matter, but because in a retrieval system the model is the last component in a chain, and it faithfully reflects whatever the earlier components handed it. A better model given bad context produces a more articulate wrong answer.

Retrieval fails silently

If the retrieval step returns the wrong chunks, generation cannot recover. The model has no way to know that the passage it was given is irrelevant; its job is to answer from context, and it will.

This failure is invisible in most deployments because nobody logs what was retrieved. The trace shows the question and the answer, and the answer looks confident, so the conclusion is "the model made it up". Frequently the model did exactly what it was told using material that should never have reached it.

The first thing I instrument is retrieval on its own. For a set of questions with known answers, was the passage containing the answer in the retrieved set at all? That single number — recall at k — separates two completely different bugs that look identical from the outside.

If retrieval recall is poor, no amount of prompt engineering helps. If retrieval recall is good and answers are still wrong, now you have a generation problem, and now changing the model is a reasonable thing to try.

Chunking is a retrieval decision disguised as preprocessing

Chunking gets treated as a preprocessing detail, and it determines what is findable.

Chunks that are too small lose the context that makes them meaningful — a paragraph that says "this is not recommended" without the sentence establishing what "this" is. Too large and the embedding averages several topics into a vector that is a good match for nothing in particular.

Two things helped more than tuning the size. Overlapping chunks so that a fact spanning a boundary appears whole in at least one. And preserving structural metadata — page, section, heading — through ingestion, both because it improves retrieval and because it is what makes real citations possible later.

Confidence is a feature, not a nuisance

When I built a multi-agent support system, the change that mattered most was not a better prompt. It was making the resolution step emit a confidence score, and routing anything below a threshold to a human.

Resolved cases averaged 0.87 confidence. Escalated ones averaged 0.62. That gap is the entire product. It turned "the bot is unreliable" into "the bot handles 65% of tickets and knows which 35% it cannot", which is a system you can actually deploy.

The escalation path mattered as much as the threshold. An escalation that dumps a customer onto a human with no context is worse than no automation — the customer repeats themselves and concludes the bot wasted their time. Every escalation carried the full trace: what was asked, what was retrieved, what was attempted, why confidence was low.

A system that knows when it does not know is worth more than one that is slightly more accurate and always certain.

Aggregate scores hide the failure that matters

On a text classification job I once got 92.76% accuracy and nearly shipped it. The per-category breakdown ran from 99.79% down to 29.22%.

The headline was carried entirely by the easy classes. One category was essentially broken, and the aggregate had smoothed it into invisibility. Any evaluation reporting a single number is hiding a distribution, and the hidden part is where your users are.

The same applies to retrieval. Mean relevance across a query set tells you very little. Relevance broken down by query type tells you that questions phrased as comparisons fail, or that anything requiring information from two documents fails, which is actionable.

What to build before tuning anything

A regression suite of real failures, collected from production rather than invented. Every time someone reports a bad answer, it becomes a test case. This is the highest value artefact in the whole system and it costs almost nothing to accumulate.

Field-level scoring rather than a single pass rate, so you can see which parts of a structured output are unreliable.

Citation accuracy scored separately from answer accuracy. A correct answer with the wrong citation is a failure for most real use cases — arguably a worse one, because it looks verified.

And a way to run the whole thing on demand, in CI, so that a prompt change produces a diff rather than a feeling. Prompts are code with no type system; the eval suite is the only thing standing between you and silent regression.

When it really is the model

Sometimes it is. If retrieval recall is high, the context window comfortably fits the material, citations check out, and the model still reasons badly across the retrieved passages — that is a capability limit, and a stronger model is the right answer.

That is a conclusion you reach after measurement, not before. It is also, in my experience, the least common of the four.