Part 5 · Applied AI Systems
Retrieval-Augmented Generation
RAG finds relevant information first, so the model can answer from sources rather than memory alone.
Key concepts
- Ingest: split documents into chunks.
- Index: embed the chunks and store them.
- Retrieve: find chunks relevant to the question, often using both keywords and embeddings.
- Rerank: order the best candidates more carefully.
- Generate: answer using the retrieved text, with citations.
- Bad retrieval leads to confident answers from the wrong source.
Retrieval-augmented generation answers from retrieved evidence.
Formal view and evidence
- Established. Retrieval and generation should be evaluated separately, using measures such as recall@k, precision, nDCG, answer faithfulness, and answer relevance. BM25 remains a strong lexical baseline.
- Debated. Optimal chunking and whether large context windows reduce the need for retrieval. Evidence depends heavily on corpus and task.
- Open problem. Reliable attribution, multi-hop retrieval, and resistance to poisoned documents (Chapter 38).
Common misconception
Common belief
"RAG eliminates hallucinations."
In practice
It reduces them when retrieval succeeds and the model stays faithful to the source.
Key takeaways
- RAG retrieves relevant evidence before generating an answer.
- Retrieval quality and answer faithfulness should be measured separately.
- RAG reduces hallucination but does not eliminate it.