Part 4 · Generative and Multimodal Models
Multimodal Models
Multimodal AI connects text, images, audio, and video in one shared understanding.
Key concepts
- Encoders convert each type of input into embeddings.
- CLIP-style training pulls matching images and captions together and pushes mismatches apart.
- Images are often split into patches, and audio is often turned into a spectrogram.
- Unified models can reason across inputs, such as a chart and a question.
- Vision-language models can still invent details that are not in an image.
Contrastive training places matching images and text close together in a shared space.
Formal view and evidence
- Established. Contrastive objectives such as InfoNCE maximize agreement for positive pairs relative to negatives. CLIP learns a shared image-text embedding from natural-language supervision.
- Debated. Whether multimodal training meaningfully addresses the symbol grounding problem, or whether physical interaction is required.
- Open problem. Long-video understanding, accurate spatial reasoning, and reducing object hallucination.
Common misconception
Common belief
"If AI describes a picture, it sees it as we do."
In practice
It processes learned representations and can confidently describe things that are not there.
Key takeaways
- Encoders map different kinds of input into comparable representations.
- Contrastive learning aligns matching pairs of data.
- Multimodal models can still describe things that are not present.