#q-learning — Reinforcement learning

Q-learning: learning a path by trial and error.

What you'll play with

Welcome to #q-learning. On screen, a 5 × 5 grid seen from above: the agent (blue sphere) waits on the grey-ringed cell, bottom left; the golden goal (+1) is top right, and a red trap (−1) sits in the middle. Each cell carries its value V(s) = max Q(s, a): all zero for now, so grey — the agent knows nothing. No one will hand it the map: it earns −0.04 per step, +1 if it reaches the goal, −1 if it falls into the trap. That is reinforcement learning: learning a path by trial and error, the way you learn to cross a house at night without turning on the lights.
Play a single step: /step. The agent picks an action (at random, since everything is still equal), plays it, collects the reward and updates one cell of the Q table with the rule Q ← Q + α·(r + γ·max Q(s′,·) − Q). Exact numbers arrive in the reply.
Run twenty episodes in one go: /episode 20. One episode = the agent restarts from the start and plays until it reaches the goal, the trap or the 100th step.
Keep going: /episode 100. Watch green spread across the grid step by step, and episode length settle toward the optimum (8 steps, the green dashed line on the curve).
One colour per cell is the value. But what does the agent do? Show its policy: /view arrows.
The γ (gamma) factor sets how much the agent cares about the future. Cut its horizon: /gamma 0.5. The Q table depends on γ, so it resets to zero.
Restart learning with this short horizon: /episode 100.
One last view: the path the agent would follow if it only obeyed its arrows, no more exploration. Type /policy — the display switches to path mode (/view path also does it).
Your turn: /grid medium (walls and two traps) or /grid maze (a 25-step corridor: count the episodes needed), /slip 0.2 for a wind that derails one action in five (the agent learns to steer clear of the trap), /epsilon 0.5 to explore more, /alpha 0.1 to learn more cautiously, /gamma 0.95 for a patient agent, then /episode 200 to see the difference; /seed 42 for other draws, /reset to restart. You have toured the lab: head back to the first channel, #neuron, to close the loop — you will see with new eyes what "learning" really means.

Channel commands

/grid <small|medium|maze> — Switch grid; the Q table resets to zero.
/episode <1..200> — Play n episodes in one go (each ≤ 100 steps) and update Q at every step.
/step — A single transition (s, a, r, s′) and its matching Q update, in numbers.
/alpha <0.01..1> — Learning rate α: how much of the error you correct at each update.
/gamma <0..0.99> — Discount factor γ: weight of the future; the Q table resets to zero.
/epsilon <0..1> — Exploration ε: probability of playing a random action rather than the best one.
/slip <0..0.3> — Wind: probability that an action slips a quarter turn; the Q table resets to zero.
/view <values|arrows|path> — What the grid shows: values, policy arrows or the greedy path.
/policy — Trace the greedy path from the start and give its length (or flag a loop).
/seed <1..9999> — Change the random draws (exploration, ties, wind); the Q table resets to zero.
/reset — Back to the small grid, α = 0.5, γ = 0.9, ε = 0.1, no wind, blank table.

Glossary

reinforcement learning: Learning by trial and error: an agent acts in an environment and only receives a reward, never the correct answer. It seeks the strategy that maximizes the sum of long-run rewards. Games (AlphaGo), robotics and LLM tuning (RLHF) all belong here.
agent, environment and reward: The reinforcement loop: the agent observes a state s, picks an action a; the environment responds with a reward r and a new state s′. Here: the cell, one of four directions, −0.04 / +1 / −1, and the arrival cell.
Q table: Table Q(s, a) that estimates, for each state and each action, the discounted return you obtain by playing a and then behaving optimally. The value of a state is V(s) = max_a Q(s, a); that is what colours the cells.
Bellman equation: Consistency relation between neighbouring values: Q(s, a) = r + γ·max_a′ Q(s′, a′). Q-learning corrects each estimate toward this target, one transition at a time: Q ← Q + α·(r + γ·max Q(s′,·) − Q). The bracketed term is the temporal-difference error.
discount factor γ: A number between 0 and 1 weighting the future: a reward received k steps ahead counts as γ^k. Close to 1, the agent is patient and values propagate far; small, it is myopic and only sees nearby rewards.
learning rate α: Share of the error corrected at each update, between 0 and 1. Large, the table learns fast but forgets everything on the smallest stroke of bad luck; small, it averages out the world's randomness (wind) at the cost of more episodes.
ε-greedy: How to pick an action: with probability ε you explore (random action), otherwise you exploit (the action of largest Q). Without exploration you never discover paths you have not tried; with too much, you never exploit what you know.
policy: A rule that maps each state to an action: what the agent does. The greedy policy plays the action of largest Q — those are the arrows and the traced path. The goal of reinforcement learning is to find the optimal policy.
off-policy and on-policy: Q-learning is off-policy: it learns the value of the greedy policy using max Q(s′,·), even when actions come from an ε-greedy exploration. SARSA is on-policy: it uses Q(s′, a′) with the action actually played next, and therefore learns the value of the exploratory policy — more cautious near traps.
DQN: Deep Q-Network: when states are too numerous for a table (pixels of an Atari game), a neural network approximates Q(s, a). Same Bellman rule, plus two tricks to stabilize training: replay memory and a frozen target network. That is the bridge between this channel and the neural networks of the lab.

Other channels in Reinforcement learning

#multi-armed-bandit — Multi-armed bandit: explore or exploit?
#q-learning — Q-learning: learning a path by trial and error.