Skip to main content

Lesson 3 — Notebooks and the workflow

Nearly all AI work starts in a notebook, and a lot of it should not stay there. This lesson explains what notebooks are genuinely good at, and the three specific ways they cause damage, because every one of them will happen to you otherwise.

What a notebook is​

A notebook is a document made of cells. Some cells hold code, some hold formatted text and images. You run a cell and its output — a number, a table, a plot — appears directly underneath it, inside the document.

Crucially, the cells share one persistent Python session. A variable defined in cell 3 is still there when you run cell 20. That session, called the kernel, keeps living between executions, which is both the entire appeal and the root of every problem.

Why this fits AI work so well​

Machine learning is empirical. You load data, look at it, try something, look at the result, adjust, try again. That loop wants three things a plain script does not give you:

  • Expensive state you keep. Loading a large dataset takes a minute. In a notebook you load it once and then run twenty experiments against it. A script reloads everything on each run.
  • Output next to the code that made it. A plot is meaningful beside the transformation that produced it, not in a separate file.
  • Narrative alongside code. Explaining why you dropped a column, in prose, immediately above the line that drops it, is the difference between an analysis someone can follow and a pile of statements.

Google Colab is a notebook hosted in your browser with the whole stack pre-installed and a free graphics processor attached. For learning, it removes every installation problem at once, which is why it is the recommended starting point.

Trap 1: hidden execution state​

This is the serious one, and it silently invalidates results.

Cells can be run in any order, any number of times, and the notebook only records the output, not the sequence that produced it. So this happens:

  1. You write a cell that filters out rows with missing prices.
  2. Later you edit that cell to filter something else, and you do not re-run the earlier cells.
  3. Your session still holds the dataframe from the first version.
  4. Every number below is computed from data that no longer corresponds to any code in the document.

The notebook looks perfectly coherent. Someone else opens it, runs it top to bottom, and gets different numbers. Nobody can tell who is wrong.

The habit that prevents it

Before trusting any result, and always before sharing: restart the kernel and run all cells from the top. If it does not reproduce, the result was an artefact of your session, not a finding. Do this at least at the end of every working session, so the gap never grows large.

A related and equally common version: a cell defines a variable, you delete the cell, and the variable stays alive in the session. Everything keeps working until you restart, at which point the notebook fails on a name that no longer exists anywhere.

Trap 2: version control​

A notebook file is JSON that contains your code, every output, and execution metadata such as cell counters. Consequently:

  • Re-running a notebook without changing a single line of code still produces a large diff, because the counters and outputs changed.
  • An embedded plot is stored as base64-encoded image data, so a one-line change can produce a diff thousands of lines long.
  • Merging two people's changes to the same notebook is genuinely unpleasant.
  • Committed outputs can contain data you did not intend to publish, which has caused real leaks of credentials and personal data.

Mitigations exist. Tools such as nbstripout remove outputs automatically before each commit, and jupytext keeps a plain-text .py twin alongside the notebook so reviews happen on readable code. Both are worth setting up on the first day of a serious project rather than the day it becomes painful.

Trap 3: notebooks are not production​

A notebook is a laboratory bench. Code that has to run every night, be tested, and be relied upon by other people belongs in ordinary Python modules.

The pattern that works, and that teams converge on independently:

The signal that it is time to extract something is simple: you have copied the same cell into a third place. At that point the logic wants to be a function in a module, imported everywhere it is needed.

What a healthy notebook looks like​

  • A title and a stated question at the top. What is this notebook trying to find out?
  • Imports in one cell, at the very top, not scattered through the document.
  • Text cells that say why, not what. The code already says what it does; it cannot say why you chose it.
  • One idea per cell. A cell that loads, cleans, trains and plots cannot be re-run selectively.
  • A conclusion at the bottom. What did you learn, and what should happen next. In three months this will be the only part you remember writing.
  • Named intermediates. df_clean and df_train beat df2 and df3, which beat overwriting df five times.
A quick self-check

If you cannot restart your kernel, run everything, and get the same numbers, you do not currently have a result. You have a session. The distinction becomes very important the first time someone asks you to defend a figure.

In three sentences​

Notebooks fit AI work because they keep expensive state, put output beside the code that made it, and let you explain your reasoning in place. Their persistent session also means cells can be run out of order, producing documents whose numbers match no version of the code, which is why restarting and running from the top is non-negotiable before trusting anything. Once logic is reused a third time it belongs in a Python module that the notebook imports, which is also what makes it testable and deployable.

Next — Lesson 4: scikit-learn →