Module 2 — Matrix product, transpose and inverse

The matrix product is the central operation of machine learning. Every layer of a neural network, every prediction of a linear model, every projection in dimensionality reduction is a matrix product. Understanding it isn't memorizing a recipe: it's seeing a matrix as a transformation applied to data.

The idea: a matrix transforms vectors

Multiplying a matrix $W$ by a vector $x$ produces a new vector. Concretely, $W$ takes the features of $x$ and recombines them:

$$y = W x$$

For a linear model with three features and one output, $W$ is a row of weights and $y$ is the predicted score: each weight says how much the corresponding feature counts. For a network layer, $W$ has several rows, and each row produces a different output. The matrix product computes all of them at once.

The dimension rule: the one thing never to forget

$$(m \times n) \cdot (n \times p) = (m \times p)$$

The inner dimensions must match (n = n), and the result takes the outer dimensions (m × p). This is error source number one. Reading it aloud helps: "m by n, times n by p, gives m by p."

import numpy as np

X = np.random.randn(100, 3)   # 100 observations, 3 features
W = np.random.randn(3, 1)     # 3 weights, 1 output
y = X @ W                      # (100, 3) @ (3, 1) = (100, 1)
y.shape                        # (100, 1) — one prediction per observation

NumPy's @ operator is the matrix product. The crucial point: the product is not commutative. X @ W and W @ X are neither equal nor even always defined. Order encodes the meaning of the transformation.

The transpose: swapping rows and columns

The transpose $X^T$ flips the matrix over its diagonal: rows become columns. A (n, d) matrix becomes (d, n).

X.shape        # (100, 3)
X.T.shape      # (3, 100)

It is used constantly to make two shapes compatible for a product. The star formula, everywhere in learning, is $X^T X$: it turns a (n, d) dataset into a (d, d) matrix summarizing the relationships between features — the core of linear regression and of the principal component analysis in module 4.

The inverse: "undoing" a transformation

The inverse $A^{-1}$ is the matrix that cancels $A$: applying one then the other returns to the starting point.

$$A^{-1} A = I$$

where $I$ is the identity matrix (ones on the diagonal, zeros elsewhere), the matrix equivalent of the number 1. The inverse lets you solve systems: the exact solution of linear regression is $\beta = (X^T X)^{-1} X^T y$.

The inverse in practice: rarely computed directly

In theory we solve with the inverse. In practice, we almost never invert explicitly: it is costly and numerically unstable. Libraries use np.linalg.solve(A, b) rather than np.linalg.inv(A) @ b, faster and more stable. Moreover, not every matrix is invertible: if two features are perfectly correlated (collinearity), the inverse doesn't exist — which is exactly what gradient descent lets us sidestep, as module 6 shows.

Why it is the heart of deep learning

A neural network chains matrix products separated by nonlinear functions:

$$h_1 = f(W_1 x), \quad h_2 = f(W_2 h_1), \quad \dots$$

Each $W_i$ is a learned transformation; each matrix product recombines the information of the previous layer. It is precisely to speed up these products — billions per second — that graphics cards (GPUs) became indispensable to AI, as the introductory course explained.

Summary

• The matrix product $Wx$ is a transformation: it recombines a vector's features; it is the basic operation of every layer and every linear model.
• Dimension rule: (m×n)·(n×p) = (m×p); inner dimensions match. The product is not commutative.
• The transpose swaps rows and columns to make shapes compatible; $X^T X$ summarizes feature relationships.
• The inverse undoes a transformation and solves systems, but is rarely computed explicitly (solve over inv) and doesn't always exist.

Next module: norms, distances and cosine similarity — how to measure the size of a vector and the closeness between observations.