Retrieval Augmented Generation

Retrieval augmented generation, or RAG, is a pattern where you look up relevant documents first and put them into the prompt. The model then answers from that material instead of from memory.

It fixes two things at once. The model can use information it was never trained on, including data from this morning, and it can cite where an answer came from.

Think of it like this. Think of an open-book exam. The student is the same. The difference is that the right pages are open in front of them, so they quote instead of recall.

How it works

• Split your documents into chunks. Whole documents rarely fit and dilute the useful part.
• Embed each chunk and store it in a vector database.
• Embed the question with the same model when a query arrives.
• Retrieve the nearest chunks. Usually the top three to ten.
• Put them in the prompt with an instruction to answer from the supplied material.

When to use it

• Your data changes. Anything current: prices, policies, documentation, tickets.
• Answers need sources. Retrieval gives you something to cite and audit.
• Your data is private. Nothing has to be baked into a model.
• Not for teaching style or format. That is fine-tuning territory, not retrieval.

Common mistakes

• "RAG replaces fine-tuning." Different jobs. RAG supplies knowledge, fine-tuning shapes behaviour, and they compose.
• "RAG guarantees correct answers." Retrieve the wrong passage and you get a confident answer grounded in the wrong thing.
• "More chunks is better." Extra context dilutes attention. A few good passages beat twenty mediocre ones.
• "It is a weekend project." The demo is. Production needs chunking strategy, retrieval tuning, and evaluation.