Module 10 — Jupyter notebooks: best practices and traps
The notebook is data science's dominant working interface, and it deserves it: nowhere else do you explore data with this fluidity. It is also the ecosystem's most criticized tool, and it deserves that too. This module gives both sides, then the practices that let you keep the fluidity without the accidents.
What the notebook does better than anything
The short loop. Run a cell, see the result, adjust, rerun — without reloading the data at every attempt. For exploration, where you do not know in advance what you are looking for, this seconds-long loop is unbeatable.
Embodied results. Tables and charts display below the code that produces them. A notebook is meant to be read: code, result, commentary, in the order of the reasoning.
Narrative mixed with computation. Markdown cells structure the analysis into a document: context, hypotheses, findings, conclusion. Well kept, a notebook is both the analysis and its report.
The trap: hidden state
The kernel keeps all variables created since it started. Cells can be executed in any order, edited then re-executed, or deleted without their variables disappearing. Consequence: the session's actual state may no longer correspond to any possible reading of the document.
The classic scenario: three hours of exploration, cells run out of order (the [17], [3], [24] execution numbers testify), a deleted cell whose variable survives — and a notebook displaying results its own code can no longer produce. Shared as is, it breaks at the recipient's end; worse, it does not break and produces different numbers.
Restarting the kernel erases all state; "Run All" replays the document top to bottom, as a reader would. A notebook that does not survive Restart & Run All is a draft, not a result. This test — at minimum before any sharing, commit or important conclusion — is the single practice that neutralizes most of the notebook's dangers.
The practices that keep a notebook healthy
A top-to-bottom structure. Imports and configuration in the first cell, data loading next, then the steps in the order of the reasoning, titled in Markdown. The notebook must be readable — and runnable — as a linear document.
Short cells with visible results. One transformation per cell, a control display (df.shape, head()) after every heavy step. Hundred-line cells cancel the format's very advantage.
No reassigning the same name across the document. The df overwritten twelve times is the first source of state confusion: after exploration, either explicit names (df_raw, df_clean), or — better — functions.
Logic migrates to modules. This is the practice that changes the scale: as soon as a treatment stabilizes, it leaves the notebook for cleaning.py (module 3), and the notebook keeps only the call:
from cleaning import load_orders, normalize
df = normalize(load_orders("data/raw/orders.csv"))
The notebook returns to what it does best — the narrative and the results — while the logic becomes testable, cleanly versionable and reusable. A mature project has short notebooks and rich modules; the reverse signals growing debt.
Notebooks and version control. The .ipynb file embeds its outputs (images included): Git diffs are unreadable and the repository swells. The standard remedies: clear outputs before committing, or the jupytext tool, which syncs every notebook with a clean .py — the .py reads well in diffs, the .ipynb remains the interface.
Notebook or script: the right tool per phase
| Phase | Tool | Why |
|---|---|---|
| Exploration, diagnosis | Notebook | Short loop, visible results |
| Stabilized processing | .py module | Testable, importable, versionable |
| Repeated execution (daily, scheduled) | Script | python pipeline.py automates; a notebook runs poorly unattended |
| Analysis report | Notebook run end-to-end | Narrative + numbers + charts in one document |
A project's natural trajectory crosses this table top to bottom: everything starts in a notebook, and what survives crystallizes into modules and scripts.
Key takeaways
- The notebook excels at exploration and narrative; its unique danger is the kernel's hidden state, which desynchronizes displayed results from visible code.
- Restart & Run All before any sharing or conclusion: the non-negotiable truth test.
- Linear structure, short cells, visible checks, no
dfoverwritten in a loop. - Stabilized logic migrates to imported modules; the notebook keeps the narrative. Outputs cleared before commits, or jupytext.
- Notebook to explore and narrate, script to repeat and automate.
One step remains: the course recap, then the 40-question exam that issues your certificate.