Module 1 — Agent, environment, reward: the framework

Supervised learning gave you a dataset of (x, y) pairs and a loss to minimize. Reinforcement learning gives you none of that. There is an agent that acts, an environment that reacts, and a scalar reward that scores the last decision. Everything else, including the very notion of a "correct answer", the agent has to discover on its own by trying things and watching what happens.

The interaction loop

At each time step t the agent observes a state s_t, chooses an action a_t, and receives from the environment a next state s_{t+1} and a reward r_{t+1}. The pair (state → action → next state, reward) repeats until the episode ends — the pole falls, the lander touches ground, the frozen lake ends in a hole or a gift.

import gymnasium as gym

env = gym.make("FrozenLake-v1", is_slippery=True)
state, info = env.reset(seed=0)

for step in range(100):
    action = env.action_space.sample()          # random policy for now
    next_state, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        state, info = env.reset()
        break
    state = next_state

The Gymnasium API is deliberately narrow: reset, step, action_space, observation_space. The same four calls will drive every environment of the course — FrozenLake here, CartPole in module 7, LunarLander in modules 9 and 10.

Return, not reward: why discounting matters

A reward is a one-step signal. What the agent optimizes is the return, the sum of future rewards from step t onward:

$$G_t = r_{t+1} + \gamma \, r_{t+2} + \gamma^2 \, r_{t+3} + \dots$$

The discount factor γ ∈ [0, 1] is the knob that says how much the agent cares about the far future. With γ = 0.99, a reward received a hundred steps from now is worth about 0.37 of the same reward received right now; with γ = 0.9, it is worth 0.0000265. Two very different agents come out of the same environment depending on that choice, and no theorem picks it for you.

Two reasons to discount that are worth spelling out. Mathematically, γ < 1 guarantees the sum converges even in an infinite-horizon problem, which the algorithms of module 4 need. Behaviorally, it encodes patience: a bank that discounts at γ = 0.5 per year values one euro tomorrow more than four euros in five years, and its policies will differ accordingly from one that discounts at 0.99.

What separates RL from supervised learning

Three differences are structural and will come back at every module.

No labels. You are not told which action was correct; you observe only the reward that followed. Assigning credit to the action that caused the reward — as opposed to the one that came ten steps earlier — is called the credit assignment problem, and it is what value functions in module 3 solve.

Data depends on the policy. In supervised learning the dataset is fixed. Here, changing the policy changes the states you visit, which changes the data you train on. This coupling breaks the i.i.d. assumption that most of your intuitions rely on, and it is the reason a naive gradient step on a deep policy can spiral.

Sequential decisions. An action is judged not by its immediate reward but by the trajectory it puts you on. A move that looks locally bad — sacrificing a piece, taking a longer route — can be globally optimal. Optimizing per-step is not optimizing overall.

Designing the reward: where projects fail

The reward function is not given by the environment in the philosophical sense — it is a modeling choice you make. And it is where most projects fail, not on the algorithm.

Two failure modes recur enough to name. Reward hacking: the agent finds a policy that maximizes the reward without doing what you wanted. A boat-racing game rewards checkpoints; an RL agent learns to loop in a small circle collecting the same three checkpoints indefinitely. The reward said "checkpoints", not "finish the race". Sparse rewards: on FrozenLake, the reward is 1 at the goal and 0 everywhere else. A random policy will almost never reach the goal, and the agent has nothing to learn from. The remedies are covered in module 6 (exploration) and module 8 (baselines), but the first-line defense is a reward that gives non-trivial signal along the way.

A "denser" reward is not always safer

It is tempting to add a small negative reward per step to encourage speed, or a bonus for being close to the goal. Both help — until they don't. Shaped rewards create their own local optima, and an agent will happily learn to exploit them. The rule is: shape sparingly, and only when the sparse reward is provably learnable.

Summary

• The RL loop is state → action → next state, reward, repeated until the episode ends; Gymnasium exposes it with reset and step.
• The agent optimizes the discounted return, not the immediate reward; the discount γ sets patience and guarantees convergence.
• RL differs from supervised learning on three points: no labels, non-i.i.d. data, sequential decisions with credit assignment.
• The reward function is a modeling choice; reward hacking and sparse rewards are the two failure modes to design against from day one.

Next module: the mathematical framework that makes this loop tractable — the Markov decision process.