Cognitive

Part 4 · Generative and Multimodal Models

Diffusion Models

Diffusion models learn to clean up noise, step by step, until a picture appears.

3 min readBuilds on Gradient Descent, Embeddings

An image emerges as noise is removed step by step.

Key concepts

  • Training gradually adds noise to images and teaches a network to predict that noise.
  • Generation runs the process in reverse, starting from random noise.
  • A text prompt guides each denoising step.
  • Guidance strengthens the prompt's influence.
  • Latent diffusion works in a compressed space, making generation much cheaper.

Training destroys images with noise. Generation learns to reverse that process.

Figure 20.1. Training destroys images with noise. Generation learns to reverse that process.
The noise schedule shifts an image from signal to noise during training.

Formal view and evidence

  • Established. The DDPM forward process has the closed form q(𝐱t∣𝐱0)=𝒩(α¯t𝐱0,(1−α¯t)𝐈). A simplified objective is 𝔼‖𝛜−𝛜θ(𝐱t,t)‖2. Score-based models connect diffusion to stochastic differential equations.
  • Established. Diffusion models can memorize and reproduce some training images, though this is uncommon relative to total generations.
  • Open problem. Faster high-quality sampling, precise control, and clear attribution of generated work to training data.

Common misconception

Common belief

"Image generators cut and paste pieces of existing pictures."

In practice

They generate pixels from learned patterns, though rare memorization of training images can occur.

Key takeaways

  • Diffusion models learn to reverse a gradual noising process.
  • Text guidance steers each denoising step.
  • Latent diffusion reduces cost by working in a compressed space.