Cognitive

Part 5 · Applied AI Systems

Retrieval-Augmented Generation

RAG finds relevant information first, so the model can answer from sources rather than memory alone.

3 min readBuilds on Embeddings, Large Language Models

Retrieval finds the passages closest in meaning to the question.

Key concepts

  • Ingest: split documents into chunks.
  • Index: embed the chunks and store them.
  • Retrieve: find chunks relevant to the question, often using both keywords and embeddings.
  • Rerank: order the best candidates more carefully.
  • Generate: answer using the retrieved text, with citations.
  • Bad retrieval leads to confident answers from the wrong source.

Retrieval-augmented generation answers from retrieved evidence.

Figure 24.1. Retrieval-augmented generation answers from retrieved evidence.
Grounding the prompt in sources makes a supported answer more likely.

Formal view and evidence

  • Established. Retrieval and generation should be evaluated separately, using measures such as recall@k, precision, nDCG, answer faithfulness, and answer relevance. BM25 remains a strong lexical baseline.
  • Debated. Optimal chunking and whether large context windows reduce the need for retrieval. Evidence depends heavily on corpus and task.
  • Open problem. Reliable attribution, multi-hop retrieval, and resistance to poisoned documents (Chapter 38).

Common misconception

Common belief

"RAG eliminates hallucinations."

In practice

It reduces them when retrieval succeeds and the model stays faithful to the source.

Key takeaways

  • RAG retrieves relevant evidence before generating an answer.
  • Retrieval quality and answer faithfulness should be measured separately.
  • RAG reduces hallucination but does not eliminate it.

Explore next