Part 5 · Applied AI Systems
Hallucination and Evaluation
AI can sound sure and still be wrong, so trustworthy systems must be tested, grounded, and checked.
Key concepts
- A hallucination is fluent output that is false or unsupported.
- Causes include missing knowledge, unclear questions, poor retrieval, and pressure to answer.
- Grounding, citations, and permission to say "I don't know" reduce the risk.
- Evals are repeatable tests run whenever a model, prompt, or data source changes.
- Outputs vary, so results should be reported as pass rates with enough test cases.
A continuous evaluation loop catches regressions before release.
Formal view and evidence
- Established. Kalai and Vempala show that calibrated generative models must hallucinate at some rate on certain arbitrary facts that are rare in training data, under their assumptions. A measured pass rate has sampling uncertainty, so small differences need confidence intervals.
- Debated. How to define and measure hallucination across open-ended tasks, and how much incentives in training and evaluation contribute.
- Open problem. Evaluation that resists Goodhart's law, where a measure stops being useful once it becomes the target.
Common misconception
Common belief
"Newer models don't hallucinate."
In practice
They can hallucinate less, but no model is perfectly reliable.
Key takeaways
- Hallucinations are fluent outputs that are false or unsupported.
- Grounding and abstention can reduce the risk.
- Evaluations should run whenever a model, prompt, or data source changes.