Cognitive

Part 2 · Neural Networks

Reinforcement Learning

Reinforcement learning discovers good behavior by trying actions and learning from the rewards that follow.

3 min readBuilds on Machine Learning Paradigms, Gradient Descent

An agent improves its policy through repeated interaction with an environment.

Key concepts

  • An agent takes actions in an environment, sees the state, and receives rewards.
  • A policy is its strategy for choosing actions.
  • It must balance exploration (trying new things) and exploitation (using what works).
  • Rewards can be delayed, so good early moves are hard to credit.
  • Badly designed rewards produce reward hacking: scoring points without achieving the real goal.

The agent-environment interaction at the center of reinforcement learning.

Figure 12.1. The agent-environment interaction at the center of reinforcement learning.
Value estimates guide which action the agent chooses next.

Formal view and evidence

  • Established. Markov decision processes formalize the problem. The optimal action-value function satisfies the Bellman equation Q*(s,a)=𝔼[r+γmaxa′Q*(s′,a′)∣s,a]. Tabular Q-learning converges under standard conditions.
  • Debated. The best combination of model-based planning and model-free learning, and how far offline RL can generalize.
  • Open problem. Sample efficiency, safe exploration, and reward specification in real-world settings.

Common misconception

Common belief

"Give AI a score and it will do what you meant."

In practice

It optimizes what you measured, which may differ from what you wanted.

Key takeaways

  • An agent learns a policy by interacting with an environment.
  • Agents must balance exploring new options with exploiting known ones.
  • A poorly specified reward can produce unintended behavior.

Explore next