Part 4 · Generative and Multimodal Models
Diffusion Models
Diffusion models learn to clean up noise, step by step, until a picture appears.
Key concepts
- Training gradually adds noise to images and teaches a network to predict that noise.
- Generation runs the process in reverse, starting from random noise.
- A text prompt guides each denoising step.
- Guidance strengthens the prompt's influence.
- Latent diffusion works in a compressed space, making generation much cheaper.
Training destroys images with noise. Generation learns to reverse that process.
Formal view and evidence
- Established. The DDPM forward process has the closed form . A simplified objective is . Score-based models connect diffusion to stochastic differential equations.
- Established. Diffusion models can memorize and reproduce some training images, though this is uncommon relative to total generations.
- Open problem. Faster high-quality sampling, precise control, and clear attribution of generated work to training data.
Common misconception
Common belief
"Image generators cut and paste pieces of existing pictures."
In practice
They generate pixels from learned patterns, though rare memorization of training images can occur.
Key takeaways
- Diffusion models learn to reverse a gradual noising process.
- Text guidance steers each denoising step.
- Latent diffusion reduces cost by working in a compressed space.