Module 10 — Project: an agent trained on a Gymnasium environment

Nine modules of building blocks. This module assembles them into a real project you can actually defend: a PPO agent that lands on the moon, evaluated correctly, saved with everything needed to reproduce it, and honest about what it does not do. LunarLander is the target because it is small enough to train on a laptop yet complex enough that seed variance is undeniable.

The project's structure

The order of steps matters, as always. Environment, then agent, then evaluation, then video — never the reverse. A common student mistake is to train for a million steps before checking whether the observation vector is what they think it is.

import gymnasium as gym
import numpy as np
from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.vec_env import VecNormalize
from stable_baselines3.common.callbacks import EvalCallback

# 1. Vectorized environments for on-policy sampling in parallel.
train_env = make_vec_env("LunarLander-v2", n_envs=8, seed=42)
train_env = VecNormalize(train_env, norm_obs=True, norm_reward=True)

# 2. Held-out evaluation environment, no reward normalization.
eval_env = make_vec_env("LunarLander-v2", n_envs=1, seed=1337)
eval_env = VecNormalize(eval_env, norm_obs=True, norm_reward=False, training=False)

Two decisions in this block deserve to be spelled out. Eight environments in parallel multiply throughput by eight without adding hyperparameters — a free acceleration that only on-policy methods (module 8) can leverage. And the evaluation environment is separate from training, with reward normalization disabled, otherwise the reward you report is not the reward the environment actually gives.

Training with seed hygiene

model = PPO(
    "MlpPolicy", train_env, verbose=1,
    learning_rate=3e-4, n_steps=1024, batch_size=64, n_epochs=4,
    gamma=0.99, gae_lambda=0.95, clip_range=0.2, ent_coef=0.01,
    seed=42,                              # reproducibility, sort of
)

eval_callback = EvalCallback(
    eval_env, best_model_save_path="./best/", log_path="./logs/",
    eval_freq=10_000, n_eval_episodes=20, deterministic=True,
)

model.learn(total_timesteps=1_000_000, callback=eval_callback)

The seed=42 above will make a single run reproducible. It will not make your reported result reproducible, because the number you actually care about is the mean over multiple seeds. Report five to ten seeds minimum, and always show the distribution — a single seed is a single sample.

Run-to-run variability: the honest story

Here is what a batch of ten seeded runs on the same LunarLander setup, same hyperparameters, looks like:

Seed	Final mean reward (100 eps, deterministic)
0	262
1	214
2	275
3	189
4	251
5	-12 (never learned)
6	258
7	233
8	244
9	267

Nine of ten seeds land above 180 (the "solved" threshold is 200 with some conventions); one seed simply failed. This is normal for PPO on LunarLander. Report both: the median or mean of successful seeds, and the fraction that succeeded at all. A paper that shows one curve is showing you one sample from a distribution that has real width.

Curves with intervals, not curves

import matplotlib.pyplot as plt
from stable_baselines3.common.results_plotter import load_results

# Load per-seed logs and align on training step.
runs = [load_results(f"./logs/seed_{i}") for i in range(10)]
steps = runs[0]["step"].values
rewards = np.stack([r["reward"].values for r in runs])

mean, std = rewards.mean(axis=0), rewards.std(axis=0)
plt.plot(steps, mean, label="mean over 10 seeds")
plt.fill_between(steps, mean - std, mean + std, alpha=0.2)
plt.axhline(200, color="k", linestyle="--", label="solved threshold")
plt.xlabel("training step"); plt.ylabel("mean reward per episode")
plt.legend(); plt.show()

The shaded band communicates uncertainty at a glance and prevents overclaiming. Anything without it should be met with skepticism, both in others' work and your own.

Recording a video

A landing that lasts twenty seconds is worth every training curve combined for a stakeholder demo. Gymnasium provides a wrapper for it:

from gymnasium.wrappers import RecordVideo

video_env = gym.make("LunarLander-v2", render_mode="rgb_array")
video_env = RecordVideo(video_env, video_folder="./videos", episode_trigger=lambda x: True)

obs, _ = video_env.reset(seed=0)
done = False
while not done:
    action, _ = model.predict(obs, deterministic=True)
    obs, reward, term, trunc, _ = video_env.step(action)
    done = term or trunc
video_env.close()

Ten videos across ten seeds are more informative than a hundred numbers. Include at least one failure in the report — a crash on seed 5 — as evidence that the pipeline was honestly evaluated, not cherry-picked.

Saving everything needed to reproduce

model.save("./ppo_lunar_lander.zip")
train_env.save("./vec_normalize.pkl")     # normalization statistics!

# Reload later:
from stable_baselines3.common.vec_env import DummyVecEnv, VecNormalize
env = DummyVecEnv([lambda: gym.make("LunarLander-v2")])
env = VecNormalize.load("./vec_normalize.pkl", env)
env.training = False; env.norm_reward = False
model = PPO.load("./ppo_lunar_lander.zip", env=env)

The VecNormalize statistics are the RL analogue of the scaler in the deep learning course: without them, the loaded model receives observations at a different scale than it learned, and its policy collapses to random. Forgetting to save them is the most common reproducibility bug in student projects.

What does not transfer to the real world

LunarLander is a physics simulator with a hand-crafted reward function, perfect state observation, and unlimited restarts. The real world offers none of these. Three concrete failure modes that every RL practitioner has to eventually confront:

Reality gap. A policy trained in a simulator with slightly wrong friction, latency or noise fails on the physical robot. Domain randomization — training on many perturbed variants of the simulator — is the standard patch, and it works surprisingly well for many tasks.

No episodic restarts. A trading bot cannot "reset" after a bad decision; a surgical robot even less. Off-policy methods with a safe exploration bound, or offline RL trained on logged data with no live interaction, become mandatory.

Reward hacking at deployment. The chatbot rewarded for user engagement learns to be sycophantic. The recommender rewarded for click-through learns to promote outrage. The reward function is a specification of the problem, and a mis-specified reward at scale is a lawsuit waiting to happen — a point of module 1 that the project makes concrete.

A checklist before you present the result

Ten seeds run, distribution reported. Videos of both a success and a failure included. VecNormalize statistics saved alongside the model. Hyperparameters logged with the run. Comparison against a random policy and a scripted baseline in the same table. Skipping any of these gives a result that looks better than it is.

Summary

• Vectorized environments multiply on-policy throughput; VecNormalize on observations is close to mandatory for continuous problems.
• Report ten seeds minimum with a shaded band; one curve is one sample from a distribution that has real width.
• Save the model, the normalization statistics, and the hyperparameters together; forgetting any of the three breaks reproducibility.
• The simulator is not the world: reality gap, no restarts, and reward hacking at scale are the three questions any serious deployment has to answer.

Final step: the recap and the 40-question exam.