Cognitive

Part 2 · Neural Networks

Recurrent Networks and Sequences

Language, music, and time unfold in order, so sequence models carry a memory from one step to the next.

3 min readBuilds on Backpropagation

A recurrent network reads a sequence one element at a time.

Key concepts

  • A recurrent neural network updates a hidden state at each step.
  • Long sequences suffer from vanishing gradients.
  • LSTMs and GRUs use gates to decide what to keep and forget.
  • Seq2seq compresses a sentence into one summary, which becomes a bottleneck.
  • That bottleneck motivated attention, which led to the Transformer.

A recurrent network carries a hidden state from one time step to the next.

Figure 11.1. A recurrent network carries a hidden state from one time step to the next.
Gated architectures preserve information across longer sequences.

Formal view and evidence

  • Established. An RNN computes 𝐡t=f(𝐡t−1,𝐱t;θ) and trains by backpropagation through time. Repeated Jacobian products make gradients shrink or grow exponentially in typical regimes. The LSTM's additive cell-state path mitigates vanishing gradients.
  • Debated. Where recurrent and state-space models can compete with Transformers, particularly on long sequences and efficiency.
  • Open problem. Architectures with long-range memory, efficient training, and strong language performance together.

Common misconception

Common belief

"Transformers made RNN ideas obsolete."

In practice

Recurrent and state-space ideas remain active research areas and are useful for streaming data.

Key takeaways

  • Recurrent networks carry a hidden state through a sequence.
  • Gated units help preserve information across longer spans.
  • The fixed-size bottleneck in early seq2seq models motivated attention.

Explore next