Part 2 · Neural Networks
Recurrent Networks and Sequences
Language, music, and time unfold in order, so sequence models carry a memory from one step to the next.
Key concepts
- A recurrent neural network updates a hidden state at each step.
- Long sequences suffer from vanishing gradients.
- LSTMs and GRUs use gates to decide what to keep and forget.
- Seq2seq compresses a sentence into one summary, which becomes a bottleneck.
- That bottleneck motivated attention, which led to the Transformer.
A recurrent network carries a hidden state from one time step to the next.
Formal view and evidence
- Established. An RNN computes and trains by backpropagation through time. Repeated Jacobian products make gradients shrink or grow exponentially in typical regimes. The LSTM's additive cell-state path mitigates vanishing gradients.
- Debated. Where recurrent and state-space models can compete with Transformers, particularly on long sequences and efficiency.
- Open problem. Architectures with long-range memory, efficient training, and strong language performance together.
Common misconception
Common belief
"Transformers made RNN ideas obsolete."
In practice
Recurrent and state-space ideas remain active research areas and are useful for streaming data.
Key takeaways
- Recurrent networks carry a hidden state through a sequence.
- Gated units help preserve information across longer spans.
- The fixed-size bottleneck in early seq2seq models motivated attention.