Module 1 — PyTorch tensors and NumPy interoperability

Course 08 approached deep learning through TensorFlow, where tensors are immutable and where a decorator switches between graph and eager modes. PyTorch takes the opposite starting point: tensors are mutable by default, everything runs eagerly, and the graph exists only for the duration of one gradient computation. This module lays the ground for the rest of the course by describing the object you will manipulate a thousand times a day — the tensor — and its close cousin the NumPy array, with which it shares more than most beginners suspect.

The Fashion-MNIST classifier that runs through this course starts here. A batch of grayscale 28-by-28 images will simply be a tensor of shape (batch, 1, 28, 28). Every abstraction added in later modules — nn.Module, DataLoader, GradScaler — will end up handling that same tensor. Reading its shape aloud, tracking its dtype and knowing which device it lives on prevents most of the puzzling errors that appear later in the course.

Creating a tensor: five constructors that cover everything

import torch

a = torch.tensor([[1.0, 2.0], [3.0, 4.0]])      # from a Python list
b = torch.zeros(3, 4)                           # filled with zeros
c = torch.ones(2, 5)                            # filled with ones
d = torch.arange(0, 10, dtype=torch.float32)    # 0, 1, ... 9
e = torch.randn(64, 1, 28, 28)                  # a fake Fashion-MNIST batch

torch.tensor builds a tensor from any Python sequence and infers the dtype. torch.zeros, torch.ones and torch.empty allocate without initialising the values, or with a constant. torch.arange and torch.linspace mirror their NumPy equivalents. torch.randn fills with a standard normal distribution, a convenient placeholder to prototype a training loop before the real data is wired in. Every constructor accepts a dtype= and a device= argument, discussed further down.

A tensor prints its shape, its dtype and, if it is not on the processor, its device. Reading a tensor description before feeding it to a layer is the single habit that catches the most bugs.

Shape, dtype, device: the three descriptors

Descriptor	What it carries	Typical trap
shape	tuple of dimensions	forgetting the batch axis, or swapping height and width
dtype	float32 by default	mixing int64 labels and float32 inputs in one operation
device	cpu, cuda:0, mps	forgetting to move labels along with inputs

The leading dimension is almost always the batch. A Fashion-MNIST batch of 32 examples has shape (32, 1, 28, 28): batch, channels, height, width. This channels-first layout is PyTorch's convention and differs from TensorFlow's channels-last default. Feeding a channels-last image to a channels-first network runs without error but yields nonsense predictions — the pixels are read as if 28 channels of 28 pixels stood in for one grayscale image.

The default floating-point dtype is float32. Labels for classification are integers, and PyTorch expects int64 for the cross-entropy loss. A common beginner mistake is to build labels as a float32 tensor and be greeted with a cryptic message about expected Long scalars. Casting is explicit: y.long() or y.to(torch.int64).

Sharing memory with NumPy: from_numpy and .numpy()

PyTorch and NumPy are so close that they can share the underlying buffer. This is the point that most tutorials handle poorly, and it deserves an unambiguous statement.

import numpy as np
import torch

array = np.arange(12, dtype=np.float32).reshape(3, 4)
tensor = torch.from_numpy(array)      # SHARES the buffer with the array

array[0, 0] = 99.0
print(tensor[0, 0])                   # 99.0: the tensor sees the write

torch.from_numpy does not copy: the tensor and the array reference the same memory. Modifying one changes the other. That is a feature — it removes copies on huge datasets — but it turns into a source of bizarre bugs when a NumPy preprocessing step is executed after a training batch has already been created from it.

Going the other way, .numpy() on a CPU tensor returns a view sharing the memory. A tensor on a GPU must first be moved back with .cpu(), otherwise the call raises. A tensor that carries a gradient graph refuses to be converted before .detach(); that is the whole point of the next module.

torch.tensor(array) copies, torch.from_numpy(array) does not

Beginners often reach for torch.tensor(array), which allocates a fresh buffer and copies. On a small array the difference is invisible; on a Fashion-MNIST training set of 60 000 images decoded into a single NumPy array, it is 47 MB copied for nothing. Use torch.from_numpy when the array is already the shape and dtype you need.

Views versus copies: why in-place breaks gradients

An operation on a tensor may return a view — a tensor sharing memory with the source — or a copy. The distinction is invisible until you start writing into the result, at which point it decides whether the source also changes.

x = torch.arange(12).reshape(3, 4)
y = x.transpose(0, 1)      # view
z = x.reshape(2, 6)        # view when possible, copy otherwise
w = x.clone()              # explicit copy, always

view, transpose, permute, unsqueeze, squeeze and standard slicing return views. contiguous produces a copy in memory order. clone copies unconditionally, and that is the safe option when a tensor must be modified without touching its source.

The in-place versions of arithmetic operations end with an underscore: x.add_(1), x.mul_(0.5), x.relu_(). They save an allocation but rewrite the source, which matters enormously as soon as gradients enter the picture. A tensor whose value has been overwritten in place after being consumed by an autograd operation raises RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation. It is a specific error worth memorising, because it is almost always caused by a stray underscore.

Broadcasting: a rule, three consequences

Broadcasting lets two tensors of different but compatible shapes participate in the same operation, by virtually replicating the smaller one to match the larger. The rule aligns shapes on the right and requires each aligned pair to be either equal, one, or missing.

images = torch.randn(32, 1, 28, 28)
mean = torch.tensor([0.5])
normalised = (images - mean) / 0.5   # broadcast: (32, 1, 28, 28) with (1,)

Consequence one, no copy is materialised: broadcasting is a memory-cheap operation. Consequence two, most silent bugs on shapes come from an unwanted broadcast, when two tensors align on shapes that were not meant to be paired. Consequence three, adding a [:, None] or an unsqueeze at the right place fixes the majority of shape errors without changing any actual data.

Read shapes aloud before every operation

When a layer complains about incompatible shapes, print the shapes of every tensor involved. Nine times out of ten the fix is a missing batch axis or a channel-first tensor fed to a channel-last consumer. Two lines of print(x.shape) cost nothing and save an hour of confusion.

In summary

• A tensor carries a shape, a dtype and a device; the leading axis is almost always the batch, and Fashion-MNIST inputs live at (batch, 1, 28, 28).
• torch.from_numpy shares memory with the NumPy array — no copy, no isolation — while torch.tensor(array) copies; choose consciously.
• View operations share memory with the source; in-place operations, marked by a trailing underscore, break autograd if applied to a tensor already consumed by a differentiated computation.
• Broadcasting virtually replicates the smaller shape to match the larger and produces no copy, which is convenient when intentional and treacherous when accidental.

Next module: autograd, the machinery that turns a chain of tensor operations into a graph you can differentiate.