Before you build RAG: four checks
2 min read
Updated
- RAG
- LLM architecture
- Evaluation
Retrieval-augmented generation solves one specific problem. Four checks that tell you whether you have that problem before you build the pipeline.
Retrieval-augmented generation is worth its complexity when answers must be grounded in written knowledge that keeps changing. It fails quietly when the documents are weak, permissions are unclear, or nobody measures whether retrieval finds the right passages. Four checks before building one.
1. Is the answer in a document or in a database?
A customer asking where their order is doesn't need semantic search over help articles. They need a lookup in the order system.
The support agent I built answers from each customer's real purchases, found by matching the ticket to commerce records through a fixed ladder of identifiers. Retrieval is used for policy only: a knowledge base of articles per brand, published as immutable revisions so any draft can be traced to the exact text behind it. Facts come from typed queries; guidance comes from documents.
2. Are the documents worth retrieving?
Stale pages, conflicting versions and articles nobody owns will sink a retrieval system before the embedding model matters. Audit the documents first: who owns each one, which version is current, and who is allowed to see it. Choose the model afterwards.
3. Can you measure retrieval on its own?
If the right passages aren't found, better generation only makes wrong answers sound more confident. Build a small set of real questions with the passages that should answer them, and measure how often retrieval returns those passages. Do this before tuning prompts or swapping models downstream.
4. What happens when the answer isn't there?
Missing evidence, stale content and questions outside the documents each need a designed response: decline, answer with a caveat, ask for the missing detail, or route to a person.
In the support agent this came from data. A survey of tickets that couldn't be matched to a purchase showed that about one in five described a purchase but gave no order number or other identifier. No retrieval or matching could reach those customers, so the right reply was a question asking for the email or order number used at purchase.
The unplanned alternative is improvisation delivered in a confident tone.
