Module 9 — Actor-critic, A2C and PPO

REINFORCE with a learned value baseline is already actor-critic in disguise. This module names the pieces, upgrades the baseline into a full critic, and moves to the algorithm that dominates modern continuous control benchmarks: proximal policy optimization (PPO). LunarLander is the environment, and Stable-Baselines3 is the reference implementation we compare against.

From baseline to critic

Take REINFORCE with a learned value function V_φ(s) as baseline. The gradient becomes:

$$\nabla_\theta J = \mathbb{E}\!\left[\sum_t \nabla_\theta \log \pi_\theta(a_t | s_t) \cdot A(s_t, a_t)\right]$$

Where the advantage A(s_t, a_t) = G_t - V_φ(s_t) measures how much better action a_t was than the average action from s_t. Two networks are now trained jointly: the actor π_θ improves the policy; the critic V_φ improves the baseline. Their two losses share the environment interactions.

The critic can be updated in the same TD flavors as before. A2C (advantage actor-critic) uses n-step returns; generalized advantage estimation (GAE) blends bootstraps at different horizons with a parameter λ ∈ [0, 1]:

$$A_t^{\text{GAE}} = \sum_{k=0}^{\infty} (\gamma \lambda)^k \, \delta_{t+k}, \quad \delta_t = r_{t+1} + \gamma V(s_{t+1}) - V(s_t)$$

λ = 0 recovers one-step TD; λ = 1 recovers Monte Carlo. Values around λ = 0.95 typically give the best bias-variance trade-off, and they are what PPO defaults to.

A2C in code, minimal

import torch
import torch.nn as nn

class ActorCritic(nn.Module):
    def __init__(self, n_obs, n_actions, hidden=128):
        super().__init__()
        self.trunk = nn.Sequential(
            nn.Linear(n_obs, hidden), nn.Tanh(),
            nn.Linear(hidden, hidden), nn.Tanh(),
        )
        self.actor = nn.Linear(hidden, n_actions)
        self.critic = nn.Linear(hidden, 1)

    def forward(self, x):
        h = self.trunk(x)
        return self.actor(h), self.critic(h).squeeze(-1)

The trunk is shared in this implementation. Sharing saves parameters but couples the two losses' scales — a common tuning headache. A safer variant uses two independent trunks, and PPO's original paper does exactly that on some benchmarks.

PPO: keeping updates small

Vanilla policy gradient has one recurring failure mode: a step that is too large collapses the policy. The gradient direction is fine, but the magnitude can push π_θ into a region where its own new data is nonsensical, and the training run never recovers. PPO is the practical answer.

The clipped surrogate objective is:

$$L^{\text{CLIP}}(\theta) = \mathbb{E}_t\!\left[\min\left(r_t(\theta) A_t, \; \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) A_t\right)\right]$$

Where r_t(θ) = π_θ(a_t | s_t) / π_{θ_old}(a_t | s_t) is the importance ratio between the updated and previous policies. The clip caps the ratio to [1 - ε, 1 + ε] (typically ε = 0.2), so no single update can move the policy by more than that factor in probability space. The min ensures the objective is a pessimistic lower bound — the gradient stops flowing once the clip is active in the wrong direction.

The intuition: PPO trades a bit of theoretical elegance for a large practical robustness margin. It has become the default in industry (OpenAI Five, GPT RLHF, most Stable-Baselines3 tutorials) precisely because it is hard to break with plausible hyperparameter choices.

The hyperparameters that actually matter

Six of PPO's dozen or so hyperparameters dominate the result on LunarLander:

Hyperparameter	Reasonable range	What breaks if wrong
Learning rate	3e-5 to 3e-4	Too high, policy collapses in a single update; too low, learning stalls
Clip range ε	0.1 to 0.3	Larger allows bigger updates but reintroduces the collapse risk
Discount γ	0.99 default	Lower for short episodes, higher makes credit assignment slower
GAE λ	0.9 to 0.97	1.0 is Monte Carlo (high variance), 0.0 is TD (high bias)
Rollout length	128 to 2048 steps	Short is more on-policy, long is more stable
Epochs per rollout	3 to 10	Too many revisits old data past the trust region

Numbers that hold across the community; deviating from them requires evidence.

Homemade PPO vs Stable-Baselines3

Implementing PPO correctly is famously subtle: an oft-cited paper lists 37 implementation details that separate a working PPO from a broken one. On LunarLander:

# Homemade PPO — main training loop, condensed.
for update in range(n_updates):
    trajectories = collect_rollout(env, model, n_steps=2048)
    advantages, returns = compute_gae(trajectories, gamma=0.99, gae_lambda=0.95)
    for epoch in range(10):
        for batch in minibatches(trajectories, batch_size=64):
            logp_new = model.log_prob(batch.states, batch.actions)
            ratio = (logp_new - batch.logp_old).exp()
            surrogate_1 = ratio * batch.advantages
            surrogate_2 = torch.clamp(ratio, 0.8, 1.2) * batch.advantages
            policy_loss = -torch.min(surrogate_1, surrogate_2).mean()
            value_loss = (model.value(batch.states) - batch.returns).pow(2).mean()
            loss = policy_loss + 0.5 * value_loss - 0.01 * model.entropy(batch.states).mean()
            optimizer.zero_grad(); loss.backward()
            nn.utils.clip_grad_norm_(model.parameters(), 0.5)
            optimizer.step()

Stable-Baselines3 does the same with a stable, tested implementation:

from stable_baselines3 import PPO

model = PPO("MlpPolicy", "LunarLander-v2", verbose=1,
            learning_rate=3e-4, n_steps=2048, batch_size=64,
            gamma=0.99, gae_lambda=0.95, clip_range=0.2)
model.learn(total_timesteps=1_000_000)

Expect a well-tuned homemade PPO to reach the LunarLander target of ~200 reward around one million steps; Stable-Baselines3 reaches it in half that with the same hyperparameters. The gap is the 37 details — orthogonal initialization, observation normalization, reward scaling, value clipping. Use the reference when you can; write your own when you have to justify the trade-offs.

Reading a PPO training curve

Reward per episode is not the only signal. Track KL(π_θ || π_θ_old) per update: if it spikes above 0.02, the clip is not enough and the learning rate should drop. Track explained_variance of the value function: if it hovers near zero, the critic is not learning and the advantages are noise. Both are exposed by Stable-Baselines3 out of the box.

Summary

• The advantage A(s, a) = G - V(s) is the low-variance signal that turns baseline REINFORCE into actor-critic.
• A2C uses n-step returns; GAE blends horizons with λ for a tunable bias-variance point.
• PPO clips the importance ratio to prevent policy collapse; it is the default of modern industrial deep RL for a reason.
• A homemade PPO teaches you why the 37 implementation details of Stable-Baselines3 exist; use the reference when performance matters.

Next module: assembling everything into a final project on LunarLander, with seeds, error bars, and the honesty about what does not transfer to the real world.