Part 3 · Language Models
Embeddings
Embeddings place words and ideas on a map where similar meanings sit close together.
Key concepts
- An embedding is a list of numbers representing meaning.
- Similar items get nearby vectors, commonly compared by cosine similarity.
- Semantic search finds related meaning without matching exact words.
- Sentences, images, and documents can be embedded too.
- Embeddings absorb patterns from their training data, including social biases.
Semantic search compares meaning by comparing vectors.
Formal view and evidence
- Established. The distributional hypothesis says words in similar contexts tend to have similar meanings. Skip-gram with negative sampling implicitly factorizes a shifted PMI matrix under certain assumptions. Cosine similarity is .
- Debated. The famous king โ man + woman โ queen result is often overstated. Standard analogy evaluation excludes the input words from the answers, which flatters the method.
- Open problem. Whether geometric proximity captures meaning or only usage, and how to remove bias without degrading useful information.
Common misconception
Common belief
"AI stores a dictionary definition for each word."
In practice
It stores positions and relationships learned from usage.
Key takeaways
- Embeddings represent meaning as vectors.
- Similarity between vectors enables search by meaning.
- Embeddings also absorb patterns and biases from their training data.