Module 2 — Stride, padding and receptive field

At the end of module 1, a Conv2D(32, (3, 3)) on a 32 by 32 CIFAR-10 image produced a 30 by 30 output. Where did the two rows and two columns go, and how do we get them back when we want to? That is the story of padding. And once we know how to control the output size at one layer, we need to think about the whole stack: how far into the input can a neuron deep in the network actually see? That is the receptive field.

The output size formula

For a convolution with a kernel of size $k$, a stride $s$ and $p$ pixels of padding added on each side, the size of the output along one axis is:

$$n_{out} \;=\; \left\lfloor \frac{n_{in} + 2p - k}{s} \right\rfloor + 1.$$

Plugging in the default Conv2D(32, (3, 3)) on our CIFAR image: $k=3$, $s=1$, $p=0$, $n_{in}=32$, so $n_{out} = 32 - 3 + 1 = 30$. The two missing rows are the ones where the 3 by 3 window would have fallen partially outside the image.

Framework naming conventions are worth committing to memory:

Padding	What it means	Output size when $s=1$
valid	no padding at all	$n_{in} - k + 1$, always smaller
same	enough padding so the output size equals $\lceil n_{in} / s \rceil$	equal to $n_{in}$ when $s=1$

same padding is not exactly $p = (k-1)/2$ on both sides when $k$ is even: Keras adds one more on the right and bottom to make the shapes work. This matters when you want two branches of a network to align.

import tensorflow as tf

x = tf.keras.Input(shape=(32, 32, 3))
a = tf.keras.layers.Conv2D(32, (3, 3), padding="valid")(x)   # 30 x 30
b = tf.keras.layers.Conv2D(32, (3, 3), padding="same")(x)    # 32 x 32
c = tf.keras.layers.Conv2D(32, (3, 3), strides=2, padding="same")(x)  # 16 x 16
print(a.shape, b.shape, c.shape)

Stride: downsample cheaply

Stride is the step of the sliding window. strides=1 slides one pixel at a time; strides=2 skips every other position and roughly halves the spatial dimensions. On CIFAR-10, one strided convolution takes 32 by 32 down to 16 by 16, then another down to 8 by 8. In module 3 we will compare strided convolution with pooling as two ways to shrink feature maps; they trade computation, parameters and inductive bias in different ways.

Stride greater than the kernel size skips pixels entirely: strides=3 with a 2 by 2 kernel misses one pixel in three, which is almost never what you want. A safe rule is $s \leq k$.

The receptive field: how far a neuron sees

A single 3 by 3 convolution has a receptive field of 3 by 3: each output neuron depends on nine input pixels. Stack two 3 by 3 convolutions, and the second layer's neuron combines nine of the first layer's neurons, each of which already covered 3 by 3 input pixels — the total effective window on the input is 5 by 5. Three stacked 3 by 3 layers reach 7 by 7. In general, for a stack of convolutions with kernels $k_i$ and strides $s_i$:

$$r_L \;=\; r_{L-1} + (k_L - 1)\, \prod_{i=1}^{L-1} s_i.$$

Strided or pooled layers multiply the receptive field growth. Every stride of 2 doubles how much of the input the next layer sees per unit of depth.

Why does this matter? Because a neuron can only respond to patterns that fall entirely inside its receptive field. A network whose deepest neurons cover 20 by 20 pixels on a 224 by 224 image will never learn a "whole cat" detector; it will at best learn a "cat texture" detector.

Compute the effective receptive field, not the theoretical one

The formula above gives the theoretical receptive field. Empirically, the influence of border pixels is much weaker than the centre, following a Gaussian shape. The effective receptive field is roughly the square root of the theoretical one for a plain stack of convolutions. That is why architectures often stack more layers than the arithmetic suggests: to compensate.

Dilated convolutions: bigger sight, same parameters

A dilated convolution (also called atrous convolution) inserts holes between the kernel weights. A 3 by 3 kernel with dilation rate 2 still has 9 weights, but they are spread over a 5 by 5 area of the input. The output size formula becomes:

$$n_{out} \;=\; \left\lfloor \frac{n_{in} + 2p - d(k-1) - 1}{s} \right\rfloor + 1,$$

where $d$ is the dilation rate. Stacking three dilated convolutions with rates 1, 2 and 4 reaches a receptive field of 15 by 15 with the same parameter count as three ordinary 3 by 3 layers. Semantic segmentation networks such as DeepLab rely on this trick to enlarge the receptive field without shrinking the feature maps, because segmentation needs a per-pixel output at full resolution.

# One dilated block, useful when you cannot afford to downsample.
x = tf.keras.Input(shape=(32, 32, 3))
y = tf.keras.layers.Conv2D(32, (3, 3), padding="same", dilation_rate=1)(x)
y = tf.keras.layers.Conv2D(32, (3, 3), padding="same", dilation_rate=2)(y)
y = tf.keras.layers.Conv2D(32, (3, 3), padding="same", dilation_rate=4)(y)

Bringing it back to CIFAR-10

On our small CIFAR-10 red thread, we build a tiny CNN with two blocks. Each block is one same convolution followed by a stride-2 convolution: the first block goes from 32 to 16, the second from 16 to 8. That gives the head enough resolution to still see the whole image, and reduces the number of activations per sample to something a laptop can chew through.

inputs = tf.keras.Input(shape=(32, 32, 3))
x = tf.keras.layers.Conv2D(32, 3, padding="same", activation="relu")(inputs)
x = tf.keras.layers.Conv2D(32, 3, strides=2, padding="same", activation="relu")(x)  # 16 x 16
x = tf.keras.layers.Conv2D(64, 3, padding="same", activation="relu")(x)
x = tf.keras.layers.Conv2D(64, 3, strides=2, padding="same", activation="relu")(x)  # 8 x 8
outputs = tf.keras.layers.GlobalAveragePooling2D()(x)                                # module 3
tf.keras.Model(inputs, outputs).summary()

A stride of 2 divides, it does not exactly halve

On an odd input size, strides=2, padding="same" rounds up: 33 becomes 17, not 16. Debug by printing every intermediate shape once and for all rather than deducing them mentally; a shape mismatch between two branches often comes from one path rounding differently from another.

In summary

• The output size of a convolution is $\lfloor (n_{in} + 2p - k)/s \rfloor + 1$; memorise it before trusting any framework's shape.
• same padding keeps the size when $s = 1$, valid shrinks by $k - 1$; strides greater than 1 downsample the map.
• The receptive field grows with depth and stride; a neuron cannot respond to a pattern larger than its receptive field, so architecture design starts by budgeting depth against the target object size.
• Dilated convolutions enlarge the receptive field with the same parameter count, at the price of a sparser sampling — the technique of choice for dense per-pixel outputs.

Next module: the pooling operations we have not yet used, and the ongoing debate between max pooling, average pooling and strided convolution for spatial reduction.