Cognitive

Part 5 · Applied AI Systems

Hallucination and Evaluation

AI can sound sure and still be wrong, so trustworthy systems must be tested, grounded, and checked.

3 min readBuilds on Probability and Uncertainty, Large Language Models, Retrieval-Augmented Generation

Fluent answers can be far more confident than they are accurate.

Key concepts

  • A hallucination is fluent output that is false or unsupported.
  • Causes include missing knowledge, unclear questions, poor retrieval, and pressure to answer.
  • Grounding, citations, and permission to say "I don't know" reduce the risk.
  • Evals are repeatable tests run whenever a model, prompt, or data source changes.
  • Outputs vary, so results should be reported as pass rates with enough test cases.

A continuous evaluation loop catches regressions before release.

Figure 28.1. A continuous evaluation loop catches regressions before release.
Evaluations reveal how a change affects different kinds of cases.

Formal view and evidence

  • Established. Kalai and Vempala show that calibrated generative models must hallucinate at some rate on certain arbitrary facts that are rare in training data, under their assumptions. A measured pass rate has sampling uncertainty, so small differences need confidence intervals.
  • Debated. How to define and measure hallucination across open-ended tasks, and how much incentives in training and evaluation contribute.
  • Open problem. Evaluation that resists Goodhart's law, where a measure stops being useful once it becomes the target.

Common misconception

Common belief

"Newer models don't hallucinate."

In practice

They can hallucinate less, but no model is perfectly reliable.

Key takeaways

  • Hallucinations are fluent outputs that are false or unsupported.
  • Grounding and abstention can reduce the risk.
  • Evaluations should run whenever a model, prompt, or data source changes.

Explore next