Module 3 — Forward pass and loss computation

The forward pass is the journey of a data point from input to prediction. It is the easy part of deep learning, and you should master it completely before tackling backpropagation, which is only this same journey read backwards.

A layer is a matrix product

Module 1 described a neuron. A layer computes all its neurons at once, and matrix notation makes that natural. For a layer $\ell$:

$$z^{(\ell)} = W^{(\ell)} a^{(\ell-1)} + b^{(\ell)}, \qquad a^{(\ell)} = f\left(z^{(\ell)}\right)$$

where $a^{(\ell-1)}$ is the previous layer's output, $W^{(\ell)}$ the weight matrix, $b^{(\ell)}$ the bias vector, and $f$ the activation applied element-wise.

Matrix shapes are the first source of errors in practice, and two minutes of checking saves many. If the previous layer has $n$ neurons and the current one $m$, then $W^{(\ell)}$ is $m \times n$, $b^{(\ell)}$ has size $m$, and the output $a^{(\ell)}$ has size $m$. The layer's parameter count is therefore $m \times n + m$.

That last formula explains model sizes. A layer of 1,000 neurons followed by another of 1,000 neurons contains a million weights on its own. It is also why fully connected layers quickly become untenable on images, and why the convolutional networks of course 10 share their weights.

Batch processing

In practice you never pass a single observation. You stack $B$ observations into a matrix $X$ of size $B \times n$, and the computation becomes:

$$Z = X W^{\top} + b$$

The result has size $B \times m$: one row per observation. Two reasons make this indispensable. The first is hardware: graphics processors are built for massively parallel matrix operations, and processing 256 observations at once is nearly as fast as processing one. The second is statistical, and module 5 will return to it: a gradient averaged over a batch is far less noisy than that of a single observation.

import numpy as np

def forward_pass(X, weights, biases):
    """Traverse a fully connected network with ReLU in hidden layers."""
    a = X
    for i, (W, b) in enumerate(zip(weights, biases)):
        z = a @ W.T + b
        last = i == len(weights) - 1
        a = z if last else np.maximum(0, z)   # linear output at the end
    return a

This code, in ten lines, is the entirety of the forward pass. All the difficulty of deep learning lies elsewhere.

The loss turns an error into a number to minimize

The network produces a prediction; the loss function measures its distance from the truth, as a single number that optimization will try to reduce. Its choice follows from the task, exactly like the output activation of the previous module — and the two must agree.

For regression, mean squared error:

$$\mathcal{L} = \frac{1}{B}\sum_{i=1}^{B}\left(y_i - \hat{y}_i\right)^2$$

It penalizes the square of the error, hence large errors heavily — useful if they are serious, harmful if the data contains outliers. Mean absolute error or the Huber loss, which is quadratic near zero and linear beyond, are then more robust.

For classification, cross-entropy. In the binary case:

$$\mathcal{L} = -\frac{1}{B}\sum_{i=1}^{B}\left[y_i \log \hat{y}_i + (1 - y_i)\log(1 - \hat{y}_i)\right]$$

It is worth understanding its behavior rather than memorizing it. If the true class is 1 and the model predicts 0.99, the term $\log(0.99)$ is nearly zero: the loss is small. If it predicts 0.01, then $\log(0.01) \approx -4.6$: the loss is large. And if the model asserts 0 with certainty when the answer is 1, the loss tends to infinity.

That is the decisive property: cross-entropy severely punishes confident errors. A model that errs while hesitating is far less penalized than one that errs categorically. This is what produces better-calibrated probabilities, and why you do not classify with squared error.

A classic numerical pitfall

Computing a softmax then its logarithm separately causes overflow: the exponential of a large score explodes, and the logarithm of zero does not exist. Every library therefore offers a fused, stabilized version — CrossEntropyLoss in PyTorch, from_logits=True in Keras. They expect the raw scores of the last layer, without softmax. Applying a softmax then this loss is a frequent mistake: the softmax is then applied twice, the model learns poorly, and nothing flags it.

Summary

• A layer computes $z = Wa + b$ then applies the activation; the layer has $m \times n + m$ parameters, which explains model sizes.
• Batch processing stacks observations: indispensable for GPU parallelism and for reducing gradient noise.
• The loss converts error into a scalar to minimize; squared error for regression, cross-entropy for classification.
• Cross-entropy punishes confident errors, producing better-calibrated probabilities; and the loss must receive raw scores, not an already-applied softmax.

Next module: backpropagation, where we walk this same path backwards to learn how much to correct each weight.