Module 7 — Deep Q-networks and experience replay
CartPole has a four-dimensional continuous state — cart position, cart velocity, pole angle, pole angular velocity. There is no Q-table to build; the number of states is uncountable. What Q-learning needs instead is a function approximator that generalizes across nearby states. A neural network is the obvious candidate, and putting one behind Q-learning is what the 2015 DQN paper did to solve Atari from pixels. This module builds a minimal DQN that solves CartPole on a laptop.
From a table to a network
Replace Q[state, action] by Q_theta(state), a network that outputs one Q-value per action:
import torch
import torch.nn as nn
class QNetwork(nn.Module):
def __init__(self, n_obs, n_actions, hidden=128):
super().__init__()
self.net = nn.Sequential(
nn.Linear(n_obs, hidden), nn.ReLU(),
nn.Linear(hidden, hidden), nn.ReLU(),
nn.Linear(hidden, n_actions), # one Q value per action
)
def forward(self, x):
return self.net(x)
The tabular update Q[s, a] += α (target - Q[s, a]) becomes a regression: minimize the squared TD error between the network's prediction and the target y = r + γ max_{a'} Q_theta(s', a'). In principle that is all it takes. In practice two additional mechanisms — the replay buffer and the target network — are what stops the whole thing from diverging.
Why naive online DQN diverges
Two independent problems undermine the naive setup.
Correlated updates. Consecutive transitions in an RL episode are highly correlated: s_t and s_{t+1} differ by one action. Feeding them one by one to a gradient step violates the i.i.d. assumption of stochastic gradient descent, and the network overfits the last trajectory then forgets the previous ones. Loss curves oscillate wildly.
Moving target. The regression target y = r + γ max_{a'} Q_theta(s', a') uses the very network that is being trained. Update Q_theta and the target moves; chase the target and it moves again. This feedback loop, without the two mechanisms below, drives the network into oscillations or divergence within a few thousand steps.
The replay buffer
Store every transition (s, a, r, s', done) in a fixed-size ring buffer (typically 100 000 for CartPole, 1 million for Atari). At each learning step, sample a random minibatch from the buffer and take a gradient step on it. Two immediate wins:
- Samples in a minibatch come from different trajectories and different times, breaking the correlation.
- Each transition is reused many times, improving sample efficiency by a factor of 10 to 100 compared to on-policy learning.
from collections import deque
import random
class ReplayBuffer:
def __init__(self, capacity=100_000):
self.buffer = deque(maxlen=capacity)
def push(self, s, a, r, s_next, done):
self.buffer.append((s, a, r, s_next, done))
def sample(self, batch_size=64):
batch = random.sample(self.buffer, batch_size)
s, a, r, s_next, done = zip(*batch)
return (
torch.tensor(np.array(s), dtype=torch.float32),
torch.tensor(a, dtype=torch.long),
torch.tensor(r, dtype=torch.float32),
torch.tensor(np.array(s_next), dtype=torch.float32),
torch.tensor(done, dtype=torch.float32),
)
def __len__(self):
return len(self.buffer)
The target network
Freeze a copy of the Q-network called Q_target, and compute the regression target with it: y = r + γ max_{a'} Q_target(s', a'). Copy Q_theta weights into Q_target every n steps (hard update, n ≈ 1000) or continuously with an exponential moving average (soft update, τ ≈ 0.005). The regression now has a stationary target for n steps at a time, and the divergence disappears.
Training loop, complete
import gymnasium as gym
import numpy as np
import copy
env = gym.make("CartPole-v1")
n_obs, n_actions = env.observation_space.shape[0], env.action_space.n
q_net, q_target = QNetwork(n_obs, n_actions), QNetwork(n_obs, n_actions)
q_target.load_state_dict(q_net.state_dict())
optimizer = torch.optim.Adam(q_net.parameters(), lr=5e-4)
buffer = ReplayBuffer()
gamma, batch_size = 0.99, 64
epsilon, eps_min, eps_decay = 1.0, 0.02, 0.995
rewards = []
for episode in range(500):
state, _ = env.reset()
total, done = 0.0, False
while not done:
# Epsilon-greedy action selection.
if random.random() < epsilon:
action = env.action_space.sample()
else:
with torch.no_grad():
q_values = q_net(torch.tensor(state, dtype=torch.float32))
action = int(q_values.argmax())
next_state, reward, term, trunc, _ = env.step(action)
done = term or trunc
buffer.push(state, action, reward, next_state, float(done))
state = next_state
total += reward
# Learning step: sample from buffer and regress on target.
if len(buffer) >= 1000:
s, a, r, s_next, d = buffer.sample(batch_size)
with torch.no_grad():
target = r + gamma * (1 - d) * q_target(s_next).max(dim=1).values
q_pred = q_net(s).gather(1, a.unsqueeze(1)).squeeze(1)
loss = nn.functional.smooth_l1_loss(q_pred, target)
optimizer.zero_grad()
loss.backward()
nn.utils.clip_grad_norm_(q_net.parameters(), 10.0)
optimizer.step()
# Soft update of the target network.
with torch.no_grad():
for p_t, p in zip(q_target.parameters(), q_net.parameters()):
p_t.data.mul_(1 - 0.005).add_(0.005 * p.data)
rewards.append(total)
epsilon = max(eps_min, epsilon * eps_decay)
CartPole is "solved" when the reward stays at 500 (the episode cap) for 100 consecutive episodes. With the code above, that typically happens between episodes 200 and 400 — with substantial run-to-run variability, a point we return to in module 10.
Double DQN: fixing the max bias
Q-learning's max overestimates, and DQN inherits that bias amplified by function approximation. Double DQN decouples action selection from action evaluation:
The online network picks the action; the target network scores it. One extra line in the code:
with torch.no_grad():
best_a = q_net(s_next).argmax(dim=1, keepdim=True)
target = r + gamma * (1 - d) * q_target(s_next).gather(1, best_a).squeeze(1)
On CartPole the improvement is small; on Atari it is substantial. It is now a default of every DQN implementation that claims to be modern.
Loss going NaN: gradient explosion; clip gradient norms and lower the learning rate. Reward plateauing at some low value: exploration decayed too fast; slow it down. Reward improving then collapsing: target updates too aggressive; increase n or lower τ. Every real DQN debugging session goes through these three, in that order.
Summary
- Deep Q-learning replaces the table by a network
Q_theta; the tabular update becomes a regression on the TD target. - The replay buffer breaks correlation between consecutive samples and enables reuse; without it, deep Q-learning oscillates.
- The target network provides a stationary regression target over short windows; without it, the moving-target loop causes divergence.
- Double DQN decouples action selection from evaluation to fix the maximization bias; a one-line change, now standard.
Next module: policy gradient methods, which optimize the policy directly rather than through Q-values.