Skip to main content

Module 9 — Virtual environments, pip and dependencies

"It works on my machine" is the most expensive sentence in data science: the notebook that ran in January breaks in June, the colleague cannot reproduce your numbers, deployment reveals incompatible versions. This whole module fits in one discipline: one project = one isolated environment = one versioned dependency file.

The problem: one Python installation for every project​

Without isolation, every pip install modifies the single installation shared by all your projects. Project A requires pandas 1.x, new project B installs pandas 2.x — and project A breaks retroactively, without a single line changed. Multiply by the dozens of transitive dependencies of a data project, and the global installation becomes a minefield where nobody knows which version of what runs what.

The solution: the virtual environment​

A virtual environment is a folder containing a Python interpreter and its own packages, independent from the rest of the machine. The standard tool, venv, ships with Python:

# At the project root — once
python -m venv .venv

# Activation — every work session
source .venv/bin/activate        # macOS / Linux
.venv\Scripts\activate           # Windows

# The prompt shows (.venv): every pip install lands HERE
(.venv) pip install pandas matplotlib seaborn

Once activated, python and pip refer to the environment's own. deactivate exits. The .venv/ folder can be recreated in a minute: it is not versioned (one line in .gitignore), unlike the file that follows.

In the data ecosystem, conda plays the same role with a historical edge on packages with compiled components, and uv makes installations radically faster. The concepts are identical; venv + pip remains the universal baseline to master first.

requirements.txt: the list that makes the project installable​

# requirements.txt
pandas==2.2.3
numpy==2.1.2
matplotlib==3.9.2
seaborn==0.13.2
jupyter==1.1.1

Anyone — including you in six months — then rebuilds the environment in two commands:

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Two schools for producing it. pip freeze > requirements.txt captures everything, transitive dependencies included: maximum reproducibility, minimum readability. The handcrafted alternative — writing only the directly used packages, pinned to their version — stays readable and is enough for most analysis projects. Modern tools (uv, Poetry) reconcile the two with a constraints file and a lock file; the principle does not change.

Pinning versions is not paranoia

pandas without a version means "whatever version the day of installation": two installations six months apart diverge, and major upgrades break real APIs. pandas==2.2.3 guarantees the identical. For a project that must last, also note the Python version itself (in the README or a .python-version file).

The full reproducibility kit​

The environment is only one of three legs. A truly reproducible project pins:

  1. The dependencies — requirements.txt versioned with the code.
  2. The randomness — every random operation gets a seed: np.random.default_rng(42), random_state=42 in scikit-learn, df.sample(n, random_state=42). Without seeds, every run produces different numbers, and "reproducing the report's result" becomes impossible.
  3. The data — at minimum: never overwrite the raw files (a read-only data/raw/ folder; transformations write to data/prepared/), and record the extraction date and source.
project/
├── .venv/               # git-ignored
├── .gitignore           # .venv/, data/
├── requirements.txt     # versioned
├── data/
│   ├── raw/             # untouchable
│   └── prepared/        # regenerable by the code
├── cleaning.py
└── analysis.ipynb

The decisive test — mental or real: a colleague clones the repo, creates the environment, runs the code: do they get your numbers? Every "no" points to a hidden dependency: an unlisted package, an absolute path to your desktop, a missing seed, a hand-edited file.

Key takeaways​

  • A Python installation shared across projects always ends in version conflicts; the virtual environment isolates each project.
  • python -m venv .venv, activate, pip install: three gestures that become automatic; .venv/ in the .gitignore.
  • A versioned requirements.txt with pinned versions (==): the project reinstalls in two commands, today as in two years.
  • Full reproducibility = frozen dependencies + random seeds + untouched raw data.
  • The criterion: can someone else get your numbers from the repository alone?

Last technical module: Jupyter notebooks — the daily working tool, its real strengths and the hidden-state trap.