Recap and final exam
Ten modules to move from a random FrozenLake walker to a PPO agent landing on the moon. Here is the course condensed, then the threads that run through it.
The course at a glance
| Module | The essential point |
|---|---|
| 1. Agent and environment | Loop state → action → next state, reward; the agent optimizes the discounted return, not the immediate reward |
| 2. Markov decision processes | An MDP is (S, A, P, R, γ); the Markov property collapses history to s_t and justifies every algorithm that follows |
| 3. Value functions and Bellman | V^π scores a policy, Q^π an action; Bellman equations tie both to themselves via one step of the MDP |
| 4. Dynamic programming, Monte Carlo | Value/policy iteration when P is known; Monte Carlo when it is not; MC is unbiased but slow |
| 5. Temporal difference, Q-learning | TD bootstraps at every step; SARSA is on-policy, Q-learning off-policy (cliff walking difference) |
| 6. Exploration | Under-exploration is a silent failure mode; epsilon decay is the default, softmax and UCB the alternatives |
| 7. Deep Q-networks | Replace the table by a network; replay buffer and target network are what stop divergence |
| 8. Policy gradient | Parameterize π_θ directly; REINFORCE with baseline reduces variance; on-policy, no replay |
| 9. Actor-critic, A2C, PPO | Advantage replaces raw return; PPO clips the importance ratio to prevent policy collapse |
| 10. Project | Ten seeds, shaded band, VecNormalize saved; the simulator is not the world |
The map of algorithms
RL algorithms can be organized on three orthogonal axes, and knowing where you sit on each is more useful than memorizing names.
Tabular vs deep. Q-learning on 16 states writes into a NumPy array; DQN on CartPole regresses a neural network. The math is the same; the machinery — replay buffer, target network, gradient clipping — appears the moment the table becomes a network.
Value vs policy. Value-based methods (Q-learning, DQN) learn Q and act greedily; policy-based methods (REINFORCE, PPO) parameterize π_θ and improve it directly. Actor-critic combines both: a critic estimates V (or Q), an actor uses it as a baseline. Continuous actions favor policy methods; small discrete action spaces favor value methods.
On-policy vs off-policy. On-policy (SARSA, REINFORCE, A2C, PPO) uses only data from the current policy; off-policy (Q-learning, DQN) reuses old data through a replay buffer. Off-policy is more sample-efficient; on-policy is more stable. Neither is universally better, and hybrid methods exist for exactly that reason.
Given a new problem, ask these three questions in order. Discrete or continuous actions? picks the family. Data cheap or expensive? picks on- vs off-policy. Model available or not? picks between model-based DP and model-free MC/TD. Ninety percent of RL algorithm selection is answering these three correctly.
The threads running through the course
Everything comes back to Bellman. Value iteration, Q-learning, DQN's regression target, the critic in A2C, the value function that gives PPO its advantage — all are instances of the same equation applied under different constraints. A student who understands module 3 can rederive most of the course.
Exploration is a first-class concern, not an afterthought. Under-exploration is silent and looks exactly like a working algorithm at reward zero. Epsilon-greedy, softmax, UCB, optimistic initialization, entropy bonuses in PPO — every algorithm has its exploration story, and forgetting to check it is the top cause of "my RL does not work" tickets.
Variance is the enemy of policy gradients. REINFORCE without baseline works but takes forever. Baseline, then advantage, then GAE, then PPO's clip — each is one more variance reducer. The story of policy gradient methods is largely the story of controlling variance without introducing too much bias.
Reproducibility in RL means seeds, not one seed. A single run reports a sample from a distribution whose width can be enormous. Ten seeds, distribution reported, is the honest minimum for any claim about "solved" or "improved".
The final exam
The exam has 40 questions covering the ten modules: framing a control problem as an MDP and diagnosing violations of the Markov property, computing values with Bellman, choosing between DP, MC and TD, tuning Q-learning and its exploration schedule, understanding why DQN needs replay and a target network, reading a policy gradient update, choosing between value-based and policy-based methods, applying PPO with realistic hyperparameters, and running a project with correct evaluation.
Several questions present situations to diagnose: a reward curve flat at zero, a DQN loss going NaN, a policy that collapses after one PPO update, a demo that looks great on the seed shown but fails on a fresh one. It is judgment that is assessed, not the recitation of formulas.
On success, your certificate of completion is issued immediately; its number is verifiable by any third party on the platform.
Take the table above and, for each row, ask yourself "how would I see that I am wrong here?". If you can say why a Q-learning run stays flat at zero, why DQN diverges without a target network, and why one PPO seed is not a result, you are ready. Good luck!
Final exam
Ready to validate this course?
40 questions drawn at random from the course bank · passing score 70% · verifiable PDF certificate issued immediately on success.
Start the examYou need to be signed in to your InSkillML account with an active subscription. You can also start the exam from My courses.