Lesson 2 — NumPy and pandas
These two libraries are underneath essentially everything else. Understanding what each solves saves you from the most common beginner mistake, which is using the wrong one for the job.
NumPy: why a list is not enough
Python already has lists, so the obvious question is why an array type was needed at all. The answer is about how memory is laid out.
A Python list of a million numbers is a million separate objects scattered across memory, with the list holding pointers to them. Each object carries type information and a reference count. Adding one to every element means a million interpreter iterations, each involving pointer chasing, type checking and object allocation.
A NumPy array of a million numbers is one contiguous block of raw bytes, all the same type. Adding one to every element is a single call into compiled code that walks the block sequentially, using the processor's vector instructions to handle several values per cycle.
| Python list | NumPy array | |
|---|---|---|
| Memory layout | scattered pointers to objects | one contiguous block |
| Element types | anything, mixed freely | one fixed type for the whole array |
| Memory for a million integers | roughly 40 MB | roughly 8 MB |
| Element-wise arithmetic | a Python loop | one compiled call |
| Typical speed on that operation | baseline | 10 to 100 times faster |
| Multi-dimensional data | nested lists, awkward | native, with a shape |
The speed gap is not a micro-optimisation. Work that takes forty seconds in a Python loop takes under a second vectorised. On a dataset you touch a hundred times during exploration, that is the difference between a productive afternoon and a wasted one.
Vectorisation, in one idea
Vectorisation means expressing an operation on the whole array at once instead of element by element. Written as an array expression, the same intent stays in the fast compiled layer:
# Slow: the loop runs in the Python interpreter
result = []
for value in temperatures:
result.append(value * 9 / 5 + 32)
# Fast: one operation on the whole array, executed in compiled code
result = temperatures * 9 / 5 + 32
The second version is also shorter and closer to how you would write the formula on paper. This is the recurring pattern of the ecosystem: the fast way is usually the readable way.
Broadcasting, in one idea
Broadcasting is NumPy's rule for combining arrays of different shapes without copying data. Subtracting a row of per-column averages from a whole table works directly: NumPy conceptually stretches the smaller shape across the larger one, without ever materialising the expanded version in memory.
This is why normalising a dataset — subtract the mean, divide by the standard deviation — is one line rather than a nested loop. It is also the source of most confusing NumPy errors, because when shapes are not compatible the message talks about dimensions rather than about your intent.
The shape is the thing to watch
Every array has a shape, a tuple of its dimensions. A grayscale image might be (128, 128). A colour image (128, 128, 3). A batch of thirty-two colour images (32, 128, 128, 3). A very large share of the errors you will hit in deep learning are shape mismatches, and the habit of printing shapes when confused pays for itself immediately.
pandas: for tables with names
NumPy is excellent for homogeneous numerical arrays. Real data is rarely that. It is a table where one column is a date, another a customer name, a third a price, with missing values scattered through it.
pandas provides the DataFrame: a table with named columns, each column with its own type, an index for the rows, and first-class handling of missing data. Underneath, the numeric columns are NumPy arrays, so you keep the speed.
What pandas is genuinely good at is the unglamorous work that consumes most of a real project:
- Loading data from CSV, Excel, JSON, SQL, Parquet
- Inspecting it: shape, types, ranges, how many values are missing and where
- Cleaning: filling or dropping missing values, fixing types, removing duplicates
- Filtering and selecting rows and columns by condition
- Grouping and aggregating: average basket by region and by month
- Joining several tables on shared keys
- Reshaping between wide and long layouts
Practitioners consistently report spending the majority of a project on data preparation rather than on modelling. That work happens in pandas. Being fluent in it affects your productivity more than knowing an extra algorithm.
Where pandas stops
pandas has a hard architectural limit worth knowing before you hit it: it loads everything into memory, and its intermediate operations often need several times the size of the data.
The practical guidance:
| Data size | Sensible tool |
|---|---|
| Up to a few hundred megabytes | pandas, comfortably |
| Around one to a few gigabytes | pandas with care: chunked reading, explicit dtypes, Parquet instead of CSV |
| Tens of gigabytes | Polars or DuckDB on a single machine, both dramatically faster and leaner |
| Hundreds of gigabytes and up | Spark, distributed across a cluster |
Two of those deserve a note. Polars offers a similar mental model with much better memory behaviour and genuine multi-core execution, and is increasingly the default choice for new work at scale. DuckDB lets you run SQL directly against files on disk, which is often the shortest path when your question is naturally a query. Spark ML covers the distributed case.
How they fit together
The division of labour is stable: pandas to prepare, NumPy to compute, a framework to model. Nearly every project moves left to right along that path.
In three sentences
NumPy replaces scattered Python objects with one contiguous typed block, which lets whole-array operations run in compiled code between ten and a hundred times faster. pandas adds named columns, mixed types and missing-value handling on top, which is why the majority of a real project happens inside it. pandas holds everything in memory, so past a few gigabytes the answer is Polars or DuckDB on one machine, and Spark beyond that.