Week 3 Notes

Deep Learning,
Week 3

Introduction to DL and GenAI, Week 3: images as data, why fully connected networks fail on them, the convolution operation, channels and parameter counts, kernel size, stride and padding with the output-size formula, 1x1 convolutions, pooling, the receptive field and how to build a CNN. Every number here was really run.

7 chapters14 worked examples~39 min read

ImagesConvolutionStride and padding1x1 convPoolingReceptive fieldCNNInterview prep

Part 1

Images and Why MLPs Fail

Weeks 1 and 2 treated every input as a flat list of numbers. That works for a customer record. It is a poor fit for a picture, whose meaning lives in how neighbouring pixels relate. This chapter shows exactly how the MLP fails on images, and that failure is the whole motivation for convolutional networks.

1.1An image is a grid of numbers

A grayscale image is one grid of brightness values from 0 (black) to 255 (white): one channel. A colour image has three such grids, R, G, B: three channels. A 100x100 RGB image is therefore 3 x 100 x 100 = 30,000 numbers. Models usually rescale to [0, 1] by dividing by 255.

main.py

import torch
# a grayscale 5x5 "image": just a grid of brightness values, 0 (black) to 255 (white)
img = torch.tensor([[0,   0,   0,   0,   0],
                    [0, 255, 255, 255,   0],
                    [0, 255,   0, 255,   0],
                    [0, 255, 255, 255,   0],
                    [0,   0,   0,   0,   0]], dtype=torch.uint8)
print("shape:", tuple(img.shape), " numbers:", img.numel())

rgb = torch.zeros(3, 100, 100)                 # an RGB image is three such grids (channels)
print("100x100 RGB image:", tuple(rgb.shape), "->", rgb.numel(), "numbers")
print("normalised to [0, 1]:", (img.float() / 255)[1].tolist())

Output

shape: (5, 5)  numbers: 25
100x100 RGB image: (3, 100, 100) -> 30000 numbers
normalised to [0, 1]: [0.0, 1.0, 1.0, 1.0, 0.0]

What we ask computers to do with images

Task	Question	Output
Classification	"is this a cat or a dog?"	one label for the image
Object detection	"where is the cat, where is the dog?"	bounding boxes + labels
Semantic segmentation	"which class is every pixel?"	a label for every pixel

1.2Why an MLP is the wrong tool

Problem 1: far too many parameters

Flatten a 100x100 RGB image and feed it to a hidden layer of 1000 units: 30 million weights in the first layer alone. A tiny 32x32 colour image into 512 units is already 1.57 million. A convolutional layer with 32 filters of 3x3x3 needs 896:

main.py

n_in = 100 * 100 * 3                     # a modest 100x100 RGB image flattened
hidden = 1000
print("inputs:", n_in)
print("first MLP layer:", f"{n_in * hidden + hidden:,}", "parameters")

n_in = 32 * 32 * 3                       # the small CIFAR-sized image of the notes
print("32x32x3 -> 512 hidden:", f"{n_in * 512 + 512:,}", "(weights 1,572,864 + 512 biases)")
print("conv layer, 32 filters of 3x3x3:", 3 * 3 * 3 * 32 + 32, "parameters (864 weights + 32 biases)")

Output

inputs: 30000
first MLP layer: 30,001,000 parameters
32x32x3 -> 512 hidden: 1,573,376 (weights 1,572,864 + 512 biases)
conv layer, 32 filters of 3x3x3: 896 parameters (864 weights + 32 biases)

Too many parameters means slow training, huge memory and, above all, overfitting: with that many free numbers the network can memorise the training pictures instead of learning what a cat is.

Problem 2: the spatial structure is thrown away

Flattening writes the pixels in one long row, so two pixels that were neighbours may end up far apart, and the network has no way to know which pixels were adjacent. To make this concrete, shuffle the pixel positions with one fixed random permutation (the same for every image: the pictures become unrecognisable noise to a human) and train again:

main.py

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
S = 12
def make(n):                                     # images with a horizontal (0) or vertical (1) bar
    X = torch.zeros(n, 1, S, S); y = torch.randint(0, 2, (n,))
    for i in range(n):
        c = torch.randint(2, 10, (1,)).item(); r = torch.randint(2, 10, (1,)).item()
        if y[i] == 0: X[i, 0, r, c - 2:c + 3] = 1
        else:         X[i, 0, r - 2:r + 3, c] = 1
    return X + 0.05 * torch.randn_like(X), y
Xtr, ytr = make(400); Xte, yte = make(300)
perm = torch.randperm(S * S)                      # ONE fixed shuffle of the pixel positions

def shuffled(X):                                  # same shuffle for every image: the picture is scrambled
    return X.flatten(1)[:, perm].reshape(-1, 1, S, S)

def fit(model, Xa, Xb):
    opt = torch.optim.Adam(model.parameters(), 0.01)
    for _ in range(200):
        opt.zero_grad(); F.cross_entropy(model(Xa), ytr).backward(); opt.step()
    return (model(Xb).argmax(1) == yte).float().mean().item()

def mlp(): return nn.Sequential(nn.Flatten(), nn.Linear(S * S, 32), nn.ReLU(), nn.Linear(32, 2))
def cnn(): return nn.Sequential(nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.AdaptiveMaxPool2d(1), nn.Flatten(), nn.Linear(8, 2))

print("MLP  on the real pictures    :", fit(mlp(), Xtr, Xte))
print("MLP  on the scrambled pictures:", fit(mlp(), shuffled(Xtr), shuffled(Xte)))
print("CNN  on the real pictures    :", fit(cnn(), Xtr, Xte))
print("CNN  on the scrambled pictures:", round(fit(cnn(), shuffled(Xtr), shuffled(Xte)), 3))

Output

MLP  on the real pictures    : 1.0
MLP  on the scrambled pictures: 1.0
CNN  on the real pictures    : 1.0
CNN  on the scrambled pictures: 0.597

The MLP is exactly as good on scrambled pictures as on real ones: it never used the layout. A CNN, which relies on locality, collapses on the scrambled images (to 0.60 here, near guessing for a two-class task). That is the clearest proof that an MLP treats an image as an unordered bag of numbers.

Problem 3: no translation invariance

A cat in the top-left corner and the same cat in the bottom-right light up different input pixels, so an MLP must learn "cat" separately for every position. Section 6.3 measures this.

1.3The two ideas behind CNNs

Locality

To understand a pixel, its neighbours matter most.
So look through a small window (3x3): a local receptive field.

Translation invariance / weight sharing

An edge looks like an edge anywhere.
So slide the same filter over the whole image, reusing its weights at every position.

Figure 1. A fully connected layer gives every input-output pair its own weight; a convolution reuses one small kernel

Precision. The notes say "translation invariance". Strictly, convolution is translation equivariant (shift the input and the feature map shifts with it); invariance (the final answer does not change) comes from stacking this with pooling and a global summary at the end. Both words are used in practice.

1.4Chapter summary

An image is a (C, H, W) grid of numbers; RGB has 3 channels.
MLP on images: too many parameters (30 million for one layer), spatial layout lost (proved by scrambling), no position-independence.
CNN fixes it with locality and weight sharing.

Next · Part 2The Convolution Operation

Part 2

The Convolution Operation

The convolution operation is simple enough to do by hand, and doing it once makes everything else in this week click.

2.1Slide, multiply, add

Figure 2. A 3x3 kernel over a 5x5 image. At the highlighted window the products add up to 27

A kernel (or filter) is a small learnable matrix, here 3x3. Place it on a 3x3 window of the image, multiply the two element by element, and add everything into one number. Slide the window one step and repeat. The grid of results is the feature map. The image below is dark on the left and bright on the right, and the kernel is built to say "dark on the left, bright on the right":

main.py

import torch, torch.nn.functional as F
img = torch.tensor([[0, 0, 0, 9, 9],
                    [0, 0, 0, 9, 9],
                    [0, 0, 0, 9, 9],
                    [0, 0, 0, 9, 9],
                    [0, 0, 0, 9, 9]], dtype=torch.float32)       # dark left, bright right: a vertical edge
k = torch.tensor([[-1, 0, 1],
                  [-1, 0, 1],
                  [-1, 0, 1]], dtype=torch.float32)               # responds to "dark on the left, bright on the right"

# slide the 3x3 window; at each stop multiply element-wise and add up
out = torch.zeros(3, 3)
for i in range(3):
    for j in range(3):
        window = img[i:i + 3, j:j + 3]
        out[i, j] = (window * k).sum()
print("by hand:\n", out)
print("window at (0,1) =", img[0:3, 1:4].tolist(), "-> sum of products =", (img[0:3, 1:4] * k).sum().item())

lib = F.conv2d(img.reshape(1, 1, 5, 5), k.reshape(1, 1, 3, 3)).squeeze()
print("F.conv2d agrees:", torch.equal(out, lib))

Output

by hand:
 tensor([[ 0., 27., 27.],
        [ 0., 27., 27.],
        [ 0., 27., 27.]])
window at (0,1) = [[0.0, 0.0, 9.0], [0.0, 0.0, 9.0], [0.0, 0.0, 9.0]] -> sum of products = 27.0
F.conv2d agrees: True

The first column of the output is 0 (that window sits wholly in the dark region: all zeros), and the other columns are 27 (each window contains the boundary). The feature map is a map of where the pattern "vertical edge, dark to bright" occurs. That is the whole idea: a filter is a pattern detector and its output says where the pattern is.

Hand-designed kernels exist: Sobel detects edges, others blur or sharpen.
In a CNN the nine numbers are learned by backpropagation, so the network invents the detectors it needs.

main.py

import torch, torch.nn.functional as F
img = torch.zeros(1, 1, 8, 8)
img[0, 0, 2:6, 2:6] = 1.0                                     # a bright square on a dark background
sobel_x = torch.tensor([[-1., 0., 1.], [-2., 0., 2.], [-1., 0., 1.]]).reshape(1, 1, 3, 3)
sobel_y = sobel_x.transpose(2, 3)
gx = F.conv2d(img, sobel_x, padding=1)[0, 0]
gy = F.conv2d(img, sobel_y, padding=1)[0, 0]
mag = (gx ** 2 + gy ** 2).sqrt()
for row in mag:
    print(" ".join("#" if v > 1.5 else "." for v in row.tolist()))
print("pixels flagged as edge:", int((mag > 1.5).sum()), "(the response hugs the border of the square, inside and out; flat regions give 0)")

Output

. . . . . . . .
. . # # # # . .
. # # # # # # .
. # # . . # # .
. # # . . # # .
. # # # # # # .
. . # # # # . .
. . . . . . . .
pixels flagged as edge: 28 (the response hugs the border of the square, inside and out; flat regions give 0)

2.2Convolution or cross-correlation?

The mathematical convolution flips the kernel before sliding it. What deep-learning libraries compute, nn.Conv2d, does not flip: it is technically a cross-correlation. The notes flag this as an exam trick. It does not matter in practice: the kernel is learned, so the network simply learns the flipped version if that is what works.

main.py

import torch, torch.nn.functional as F
x = torch.tensor([[1., 2., 3., 4.]]).reshape(1, 1, 1, 4)
k = torch.tensor([[1., 0., -1.]]).reshape(1, 1, 1, 3)       # an asymmetric kernel

pytorch = F.conv2d(x, k).flatten()                            # what nn.Conv2d computes
flipped = F.conv2d(x, k.flip(-1)).flatten()                   # true mathematical convolution flips the kernel first
print("PyTorch 'convolution' (no flip):", pytorch.tolist())
print("true convolution (kernel flipped):", flipped.tolist())
print("(they differ: the sign flips; for a LEARNED kernel it makes no difference)")

Output

PyTorch 'convolution' (no flip): [-2.0, -2.0]
true convolution (kernel flipped): [2.0, 2.0]
(they differ: the sign flips; for a LEARNED kernel it makes no difference)

2.3Many filters, many channels

Figure 3. A colour input has 3 channels, so each filter is 3x3x3 and produces one feature map; 32 filters make 32 maps

With an RGB input, one filter is really a 3x3x3 block: it has a 3x3 slice for each input channel, the three products-sums are added into one number per position. So one filter gives one feature map, and $K$ filters give $K$ feature maps, the next layer's $K$ channels. The parameter count of a convolution layer:

\#\text{params}=\underbrace{K_h\,K_w\,C_{in}\,C_{out}}_{\text{weights}}+\underbrace{C_{out}}_{\text{biases}}

main.py

import torch, torch.nn as nn
conv = nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3)
print("weight shape:", tuple(conv.weight.shape), " = (out_channels, in_channels, kH, kW)")
print("bias shape  :", tuple(conv.bias.shape))
print("parameters  :", sum(p.numel() for p in conv.parameters()), " = 3*3*3*32 + 32")

x = torch.randn(8, 3, 32, 32)                     # a batch of 8 RGB images
print("input  :", tuple(x.shape))
print("output :", tuple(conv(x).shape), "<- 32 feature maps, each 30x30 (no padding)")

for cin, cout, k in [(1, 8, 3), (8, 16, 3), (64, 128, 3), (64, 64, 1)]:
    c = nn.Conv2d(cin, cout, k)
    print(f"Conv2d({cin:3d},{cout:3d},k={k}) parameters = {sum(p.numel() for p in c.parameters()):>7,}  (= {k}*{k}*{cin}*{cout} + {cout})")

Output

weight shape: (32, 3, 3, 3)  = (out_channels, in_channels, kH, kW)
bias shape  : (32,)
parameters  : 896  = 3*3*3*32 + 32
input  : (8, 3, 32, 32)
output : (8, 32, 30, 30) <- 32 feature maps, each 30x30 (no padding)
Conv2d(  1,  8,k=3) parameters =      80  (= 3*3*1*8 + 8)
Conv2d(  8, 16,k=3) parameters =   1,168  (= 3*3*8*16 + 16)
Conv2d( 64,128,k=3) parameters =  73,856  (= 3*3*64*128 + 128)
Conv2d( 64, 64,k=1) parameters =   4,160  (= 1*1*64*64 + 64)

Two facts to remember: the count depends on the kernel and channels, never on the image size (the same 896 weights run on a 32x32 or a 1000x1000 image), and a common exam slip is forgetting the $+C_{out}$ biases.

2.4What the layers learn: a hierarchy

Nobody tells the network what filters to build. Train a tiny CNN to tell horizontal bars from vertical bars (two filters, plain gradient descent) and then print the learned kernels:

main.py

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
S = 12
def make(n):
    X = torch.zeros(n, 1, S, S); y = torch.randint(0, 2, (n,))
    for i in range(n):
        c = torch.randint(2, 10, (1,)).item(); r = torch.randint(2, 10, (1,)).item()
        if y[i] == 0: X[i, 0, r, c - 2:c + 3] = 1         # horizontal bar
        else:         X[i, 0, r - 2:r + 3, c] = 1         # vertical bar
    return X + 0.05 * torch.randn_like(X), y
X, y = make(400)

net = nn.Sequential(nn.Conv2d(1, 2, 3, padding=1, bias=False), nn.ReLU(), nn.AdaptiveMaxPool2d(1), nn.Flatten(), nn.Linear(2, 2))
opt = torch.optim.Adam(net.parameters(), 0.02)
for _ in range(150):
    opt.zero_grad(); F.cross_entropy(net(X), y).backward(); opt.step()
print("training accuracy:", (net(X).argmax(1) == y).float().mean().item())
for i, k in enumerate(net[0].weight.detach()):
    print(f"learned filter {i}:")
    print(k[0].round(decimals=1))

Output

training accuracy: 1.0
learned filter 0:
tensor([[-0.5000, -0.9000, -1.0000],
        [ 1.0000,  1.3000,  1.3000],
        [-0.5000, -0.7000, -0.8000]])
learned filter 1:
tensor([[-0.1000, -0.5000,  1.0000],
        [-0.6000, -0.6000,  1.0000],
        [-0.1000, -0.5000,  1.0000]])

Filter 0 has a bright middle row and dark rows above and below: a horizontal-line detector. Filter 1 has a bright right column and dark left: a vertical-edge detector. Edge detectors emerged from nothing but the loss. In real networks this repeats up the stack: early layers find edges and corners, middle layers combine them into textures and parts, deep layers respond to whole objects (eyes, wheels, faces).

2.5Chapter summary

Convolution = slide a small kernel, multiply element-wise, add: output says where the pattern is.
Conv2d is cross-correlation (no flip); irrelevant for learned kernels.
Params $=K_hK_wC_{in}C_{out}+C_{out}$ , independent of the image size.
Filters are learned; layers build edges, then parts, then objects.

Next · Part 3Kernel Size, Stride, Padding

Part 3

Kernel Size, Stride, Padding

Three dials control how a filter moves and how big the result is: kernel size, stride and padding. Together they fix the output size, a calculation that is asked in almost every exam on this week.

3.1Kernel size

3x3 is the standard (VGG, ResNet). A bigger kernel sees more context but costs more parameters ( $K^2$ per channel pair) and, as Chapter 4 shows, a stack of small kernels reaches the same field for less. Odd sizes (3, 5, 7) have a centre pixel, which keeps padding symmetric.

3.2Stride and padding

Figure 4. Stride is the step between window positions; padding adds zeros around the border

Stride $S$ : how many pixels the window jumps. $S=1$ overlaps heavily; $S=2$ roughly halves the output (a learned way to downsample).
Padding $P$ : zeros added all around. "valid" = no padding: the output shrinks. "same" = pad so that the output has the input's size (for $S=1$ ). Padding also keeps the border pixels from being under-used.

3.3The output size formula

H_{out}=\left\lfloor\frac{H_{in}+2P-K}{S}\right\rfloor+1,\qquad W_{out}\text{ likewise}

Why: the window's top-left corner can start at positions $0, S, 2S,\dots$ as long as the window still fits in the padded width $H+2P$ , i.e. the last start $\le H+2P-K$ . Counting those starts gives the formula. The notes write it without the floor; the floor matters whenever $(H+2P-K)/S$ is not a whole number, and PyTorch rounds down (the window that does not fully fit is dropped). Verified against PyTorch over 240 combinations of $H, K, P, S$ , with the worked examples:

main.py

import torch, torch.nn as nn
def out_size(H, K, P, S):
    return (H + 2 * P - K) // S + 1               # floor division

checked, bad = 0, 0
for H in (7, 8, 28, 32, 33):
    for K in (1, 3, 5, 7):
        for P in (0, 1, 2, 3):
            for S in (1, 2, 3):
                if H + 2 * P < K:
                    continue
                real = nn.Conv2d(1, 1, K, stride=S, padding=P)(torch.zeros(1, 1, H, H)).shape[-1]
                checked += 1
                bad += real != out_size(H, K, P, S)
print("combinations checked:", checked, " mismatches:", bad)
for H, K, P, S in [(32, 5, 0, 1), (28, 3, 1, 1), (28, 3, 0, 2), (32, 3, 0, 2), (7, 3, 0, 2)]:
    print(f"H={H:2d} K={K} P={P} S={S}  ->  (H+2P-K)/S+1 = {(H + 2 * P - K) / S + 1:g}  ->  floor {out_size(H, K, P, S)}")

Output

combinations checked: 240  mismatches: 0
H=32 K=5 P=0 S=1  ->  (H+2P-K)/S+1 = 28  ->  floor 28
H=28 K=3 P=1 S=1  ->  (H+2P-K)/S+1 = 28  ->  floor 28
H=28 K=3 P=0 S=2  ->  (H+2P-K)/S+1 = 13.5  ->  floor 13
H=32 K=3 P=0 S=2  ->  (H+2P-K)/S+1 = 15.5  ->  floor 15
H= 7 K=3 P=0 S=2  ->  (H+2P-K)/S+1 = 3  ->  floor 3

The notes' example (32x32, K=5, P=0, S=1) gives 28x28. Look at the two stride-2 rows: 13.5 and 15.5 are not integers, so PyTorch drops the leftover and gives 13 and 15. Quick rules: $S=1,P=0\Rightarrow H-K+1$ ; $S=1$ and $P=(K-1)/2\Rightarrow$ same size; $S=2\Rightarrow\approx H/2$ .

"Same" padding: we want $H_{out}=H$ at $S=1$ , so $H+2P-K+1=H$ , giving $P=(K-1)/2$ : 1 for K=3, 2 for K=5, 3 for K=7.

main.py

import torch, torch.nn as nn
x = torch.zeros(1, 1, 28, 28)
for K in (3, 5, 7):
    P = (K - 1) // 2
    out = nn.Conv2d(1, 1, K, padding=P)(x).shape[-1]
    print(f"K={K}: padding P=(K-1)/2={P}  output {out}x{out}  (same as input: {out == 28})")
print("padding='same' does it for you:", tuple(nn.Conv2d(1, 4, 5, padding="same")(x).shape))
print("padding=0 ('valid'), K=5:", tuple(nn.Conv2d(1, 4, 5)(x).shape), " <- each conv shrinks the image")

Output

K=3: padding P=(K-1)/2=1  output 28x28  (same as input: True)
K=5: padding P=(K-1)/2=2  output 28x28  (same as input: True)
K=7: padding P=(K-1)/2=3  output 28x28  (same as input: True)
padding='same' does it for you: (1, 4, 28, 28)
padding=0 ('valid'), K=5: (1, 4, 24, 24)  <- each conv shrinks the image

3.4The 1x1 convolution

A 1x1 kernel looks at one pixel but across all its channels. It cannot see neighbours; it mixes channels (a learned weighted combination of the feature maps at every position), and it can shrink or grow the number of channels without touching height or width. Its cost is $C_{in}C_{out}$ (+ biases), and it is mathematically a Linear layer applied independently at every pixel:

main.py

import torch, torch.nn as nn
torch.manual_seed(0)
x = torch.randn(2, 64, 10, 10)                        # 64 channels on a 10x10 grid
conv = nn.Conv2d(64, 16, kernel_size=1)
print("1x1 conv:", tuple(x.shape), "->", tuple(conv(x).shape), " height and width untouched")
print("parameters:", sum(p.numel() for p in conv.parameters()), " = 64*16 + 16")

# a 1x1 conv is the same linear layer applied at every pixel's channel vector
lin = nn.Linear(64, 16)
lin.weight.data = conv.weight.data.reshape(16, 64); lin.bias.data = conv.bias.data
per_pixel = lin(x.permute(0, 2, 3, 1)).permute(0, 3, 1, 2)
print("identical to a Linear layer at each pixel:", torch.allclose(conv(x), per_pixel, atol=1e-5))

# why use it: shrink channels before an expensive 3x3 conv
direct = nn.Conv2d(256, 256, 3, padding=1)
reduced = nn.Sequential(nn.Conv2d(256, 64, 1), nn.Conv2d(64, 64, 3, padding=1), nn.Conv2d(64, 256, 1))
count = lambda m: sum(p.numel() for p in m.parameters())
print(f"3x3, 256->256 directly: {count(direct):,}   via 1x1 bottleneck (256->64->64->256): {count(reduced):,}")

Output

1x1 conv: (2, 64, 10, 10) -> (2, 16, 10, 10)  height and width untouched
parameters: 1040  = 64*16 + 16
identical to a Linear layer at each pixel: True
3x3, 256->256 directly: 590,080   via 1x1 bottleneck (256->64->64->256): 70,016

The last line is why 1x1 convolutions are called a "network-in-network" and are used as bottlenecks in Inception and ResNet (Week 4): squeeze 256 channels to 64, do the expensive 3x3 there, expand back. Parameters fall from 590,080 to 70,016, a factor of 8.4.

3.5Chapter summary

$H_{out}=\lfloor(H+2P-K)/S\rfloor+1$ (the floor matters); same padding $P=(K-1)/2$ at $S=1$ .
Stride 2 halves the size; "valid" shrinks, "same" preserves.
1x1 conv mixes channels and is the bottleneck tool.

Next · Part 4Pooling and the Receptive Field

Part 4

Pooling and the Receptive Field

Convolutions find local patterns. To recognise a whole object the network must zoom out: summarise regions into smaller maps, so that later filters see a larger part of the picture. Pooling does the summarising, and the receptive field measures how much of the picture each unit can see.

4.1Pooling

Figure 5. Each 2x2 window becomes one number: its maximum, or its average

A pooling layer slides a window (usually 2x2, stride 2) over each channel separately and replaces it with one summary: max (keep the strongest response, "the feature was found somewhere in this window") or average. It has no learnable parameters, and halves height and width.

main.py

import torch, torch.nn as nn
x = torch.tensor([[1., 3., 2., 0.],
                  [4., 6., 1., 1.],
                  [0., 2., 9., 5.],
                  [1., 1., 3., 7.]]).reshape(1, 1, 4, 4)
mp = nn.MaxPool2d(2)                         # 2x2 windows, stride 2 by default
ap = nn.AvgPool2d(2)
print("max pool:\n", mp(x).squeeze())
print("avg pool:\n", ap(x).squeeze())
print("parameters in a pooling layer:", sum(p.numel() for p in mp.parameters()))
print("shape:", tuple(x.shape), "->", tuple(mp(x).shape), "(height and width halved)")

Output

max pool:
 tensor([[6., 2.],
        [2., 9.]])
avg pool:
 tensor([[3.5000, 1.0000],
        [1.0000, 6.0000]])
parameters in a pooling layer: 0
shape: (1, 1, 4, 4) -> (1, 1, 2, 2) (height and width halved)

Fewer values downstream: cheaper computation, fewer parameters in the final linear layer.
Bigger receptive field: the next 3x3 filter now covers a 6x6 patch of the original.
Some shift tolerance: a feature that moves within a window gives the same pooled output.

The last point is only approximate. A one-pixel shift inside a window vanishes; a shift that crosses a window border changes the output:

main.py

import torch, torch.nn as nn
pool = nn.MaxPool2d(2)
a = torch.zeros(1, 1, 6, 6); a[0, 0, 2, 2] = 5.0            # a bright pixel at (2, 2)
b = torch.zeros(1, 1, 6, 6); b[0, 0, 3, 3] = 5.0            # the same pixel shifted by 1
print("inputs equal:", torch.equal(a, b))
print("after 2x2 max pool, equal:", torch.equal(pool(a), pool(b)), " <- small shifts inside a window disappear")
c = torch.zeros(1, 1, 6, 6); c[0, 0, 1, 1] = 5.0            # shifted by 1 the other way: crosses a window border
print("shift across a window border, equal:", torch.equal(pool(a), pool(c)), " <- pooling is only approximately invariant")

Output

inputs equal: False
after 2x2 max pool, equal: True  <- small shifts inside a window disappear
shift across a window border, equal: False  <- pooling is only approximately invariant

4.2The receptive field

Figure 6. Stacking layers widens what one top unit can see: three 3-wide layers see 7 inputs

The receptive field of a unit is the region of the input that can influence it. One 3x3 layer sees 3; stacking a second 3x3 layer, each of its units sees three neighbours that each see three: $3+2=5$ . Each extra stride-1 3x3 layer adds 2. A stride or pool makes later layers' steps bigger, so the field grows faster. The recurrence is $r\leftarrow r+(k-1)\cdot j$ , $j\leftarrow j\cdot s$ ( $j$ = distance between neighbouring units, measured in input pixels):

main.py

def receptive_field(layers):
    """layers: list of (kernel, stride). Returns the receptive field size after each layer."""
    rf, jump, out = 1, 1, []
    for k, s in layers:
        rf = rf + (k - 1) * jump        # each layer adds (k-1) steps of the current jump
        jump = jump * s
        out.append(rf)
    return out

print("one 3x3 conv           :", receptive_field([(3, 1)]))
print("two 3x3 convs          :", receptive_field([(3, 1)] * 2), " <- equals one 5x5")
print("three 3x3 convs        :", receptive_field([(3, 1)] * 3), " <- equals one 7x7")
print("conv3, pool2, conv3    :", receptive_field([(3, 1), (2, 2), (3, 1)]))
print("conv3, pool2 x3 blocks :", receptive_field([(3, 1), (2, 2)] * 3))

C = 64
print()
print(f"parameters with {C} channels in and out (no bias):")
print("  three 3x3 convs:", 3 * (3 * 3 * C * C), "   one 7x7 conv:", 7 * 7 * C * C, "   (same 7x7 receptive field)")
print("  two 3x3 convs  :", 2 * (3 * 3 * C * C), "   one 5x5 conv:", 5 * 5 * C * C, "   (same 5x5 receptive field)")

Output

one 3x3 conv           : [3]
two 3x3 convs          : [3, 5]  <- equals one 5x5
three 3x3 convs        : [3, 5, 7]  <- equals one 7x7
conv3, pool2, conv3    : [3, 4, 8]
conv3, pool2 x3 blocks : [3, 4, 8, 10, 18, 22]

parameters with 64 channels in and out (no bias):
  three 3x3 convs: 110592    one 7x7 conv: 200704    (same 7x7 receptive field)
  two 3x3 convs  : 73728    one 5x5 conv: 102400    (same 5x5 receptive field)

VGG's insight, as numbers

Two 3x3 layers have the receptive field of one 5x5, but with 64 channels they cost 73,728 weights against 102,400 (28% fewer); three 3x3 layers match a 7x7 for 110,592 against 200,704 (45% fewer). They also add extra non-linearities (a ReLU between each pair), so they can represent more. That is why modern networks stack 3x3 convolutions instead of using big kernels.

Note on the notes. The notes count "3x3: 9+9 = 18 parameters (two layers) against 5x5: 25". That is per single input and output channel; with

C

channels the real ratio is

18C^2:25C^2

, the 28% saving above. Same conclusion, the code gives the real numbers.

4.3Chapter summary

Pooling: window summary, no parameters, halves H and W; max keeps the strongest response.
Approximate shift tolerance only; crossing a window border changes the output.
Receptive field grows with depth and strides; stacked 3x3s match big kernels with fewer weights.

Next · Part 5Building a CNN

Part 5

Building a CNN

Now assemble the pieces into a complete network and see the payoff on a task where the position of the object changes.

5.1The standard recipe

Stage	Layers	What it does
Feature extractor	[Conv -> ReLU -> Pool] repeated	turns pixels into feature maps; spatial size shrinks, channels grow
Flatten	Flatten	feature maps into one vector
Classifier	Linear -> (ReLU) -> Linear	vector into class scores (logits)

Trace the shapes of a small CNN for 28x28 grayscale images. The rule to watch: height and width fall, channels rise, until the last Linear layer, whose input size is the number of values after Flatten ( $16\cdot7\cdot7=784$ ):

main.py

import torch, torch.nn as nn
net = nn.Sequential(
    nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(),      # 28x28 -> 28x28, 8 channels
    nn.MaxPool2d(2),                               # -> 14x14
    nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(),     # -> 14x14, 16 channels
    nn.MaxPool2d(2),                               # -> 7x7
    nn.Flatten(),                                  # -> 16*7*7 = 784
    nn.Linear(16 * 7 * 7, 10),
)
x = torch.zeros(1, 1, 28, 28)
print(f"{'input':22s}", tuple(x.shape))
for layer in net:
    x = layer(x)
    p = sum(q.numel() for q in layer.parameters())
    print(f"{layer.__class__.__name__:22s}", f"{str(tuple(x.shape)):18s}", f"params {p}")
print("total parameters:", sum(q.numel() for q in net.parameters()))

Output

input                  (1, 1, 28, 28)
Conv2d                 (1, 8, 28, 28)     params 80
ReLU                   (1, 8, 28, 28)     params 0
MaxPool2d              (1, 8, 14, 14)     params 0
Conv2d                 (1, 16, 14, 14)    params 1168
ReLU                   (1, 16, 14, 14)    params 0
MaxPool2d              (1, 16, 7, 7)      params 0
Flatten                (1, 784)           params 0
Linear                 (1, 10)            params 7850
total parameters: 9098

Note where the parameters are. The two convolutions have only 80 and 1,168; the single final Linear has 7,850, 86% of the total. Convolutions are cheap in parameters but expensive in computation (they run at every position); the classifier is the reverse. (The earlier MLP in Week 2 put 92% of its parameters in its first layer.)

5.2A common mistake: the Flatten size

The input of the first Linear must equal channels x height x width at that point. Change an image size, a stride or a padding and you must recompute it, or you get the mat1 and mat2 shapes cannot be multiplied error from Week 2. Tracing shapes with a dummy tensor (as above) is the reliable fix.

5.3Where the CNN wins: when the object moves

The decisive experiment. Images contain a horizontal bar, a vertical bar or a plus. In training, shapes only ever appear in the left part of the image. At test time we check them on the left (seen before) and on the right (a place the network never saw a shape). One MLP and one tiny CNN, same data, same training:

main.py

import torch, torch.nn as nn
torch.manual_seed(0)
S = 16
def make(n, side):
    """horizontal bar (0), vertical bar (1) or plus (2) placed in the LEFT or RIGHT part of the image."""
    X = torch.zeros(n, 1, S, S); y = torch.randint(0, 3, (n,))
    for i in range(n):
        c = torch.randint(2, 6, (1,)).item() if side == "left" else torch.randint(9, 13, (1,)).item()
        r = torch.randint(3, 13, (1,)).item()
        k = y[i].item()
        if k == 0:   X[i, 0, r, c - 2:c + 3] = 1
        elif k == 1: X[i, 0, r - 2:r + 3, c] = 1
        else:        X[i, 0, r, c - 2:c + 3] = 1; X[i, 0, r - 2:r + 3, c] = 1
    return X + 0.1 * torch.randn_like(X), y

Xtr, ytr = make(600, "left")                      # training: shapes only ever appear on the left
Xl, yl = make(300, "left"); Xr, yr = make(300, "right")

def train(model, epochs=40):
    opt = torch.optim.Adam(model.parameters(), 1e-3); loss_fn = nn.CrossEntropyLoss()
    for _ in range(epochs):
        perm = torch.randperm(len(Xtr))
        for i in range(0, len(Xtr), 50):
            idx = perm[i:i + 50]
            opt.zero_grad(); loss_fn(model(Xtr[idx]), ytr[idx]).backward(); opt.step()
    model.eval()
    with torch.no_grad():
        return [(model(X).argmax(1) == y).float().mean().item() for X, y in ((Xl, yl), (Xr, yr))]

mlp = nn.Sequential(nn.Flatten(), nn.Linear(S * S, 64), nn.ReLU(), nn.Linear(64, 3))
cnn = nn.Sequential(nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
                    nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.AdaptiveMaxPool2d(1), nn.Flatten(), nn.Linear(16, 3))
for name, m in (("MLP", mlp), ("CNN", cnn)):
    left, right = train(m)
    print(f"{name}: {sum(p.numel() for p in m.parameters()):6,} parameters   test on LEFT {left:.2f}   test on RIGHT (never seen there) {right:.2f}")
print("chance level is 0.33")

Output

MLP: 16,643 parameters   test on LEFT 0.93   test on RIGHT (never seen there) 0.35
CNN:  1,299 parameters   test on LEFT 1.00   test on RIGHT (never seen there) 1.00
chance level is 0.33

Figure 7. Accuracy when the shapes are in the training region and when they move

The MLP scores 93% on the left, then 35% on the right: chance level. Having learned "pixel 37 is bright means vertical bar", it has nothing to say about pixel 200. The CNN is perfect in both places, with 13 times fewer parameters (1,299 against 16,643), because its filters learned "this pattern" and apply it everywhere. The global max pool at the end (AdaptiveMaxPool2d) is what turns that equivariance into an answer that does not depend on position.

Fairness note. This is a deliberately translation-heavy test, to isolate the effect. On data where objects already appear everywhere in training, the gap between a small MLP and a CNN is smaller, though CNNs still win clearly on real photographs, where an MLP needs far more parameters.

5.4Chapter summary

Recipe: [Conv, ReLU, Pool] x n, Flatten, Linear. H and W shrink, channels grow.
Trace shapes with a dummy tensor; the first Linear's input = C x H x W after Flatten.
Measured: MLP 35% (chance) on unseen positions, CNN 100% with 13x fewer parameters.

Next · InterviewInterview Question Bank

Interview

Interview Question Bank

What is likely to be asked from Week 3, with 30-second answers, deeper explanations, a lecture-or-extra tag on each question, and code you can run.

Week 3 mixes numerical questions (output size, parameter counts, receptive field), which reward carefulness, with why questions (why MLPs fail, why 3x3 stacks), which reward a clear sentence. Each answer starts with a 30-second version. Questions are tagged From the notes or Extra.

1Definition

One clear sentence: what it is.

2Intuition or tiny example

Why it works, in one picture or three lines.

3The failure it prevents

What goes wrong without it.

6.1Interview heat-map: what is most likely

Very likely7 topics

Why MLPs fail for images"Why are MLPs unsuitable for images?"Notes 1.2

The convolution operation"Explain the convolution operation."Notes 2.1

Output size and parameter counts"Compute the output size and the parameter count of a conv layer."Notes 3.3, 2.3

Weight sharing and locality"What are weight sharing and locality, and why do they help?"Notes 1.3

3x3 stacks versus big kernels"Why use two 3x3 convolutions instead of one 5x5?"Notes 4.2

Pooling"What does pooling do? Does it have parameters?"Notes 4.1

Same and valid padding"What is padding and why do we use it?"Notes 3.2-3.3

Likely5 topics

1x1 convolution"What is a 1x1 convolution used for?"Notes 3.4

Cross-correlation vs convolution"Is deep-learning convolution real convolution?"Notes 2.2

Invariance vs equivariance"Translation invariance or equivariance?"Notes 1.3

Channels and filters"How does a filter work on an RGB image?"Notes 2.3

Flatten and the shapes"My Linear layer after the conv layers gives a shape error."Notes 5.2

Sometimes2 topics

Sobel and hand-made kernels"What does a Sobel filter do?"Notes 2.1

Stride vs pooling"Stride 2 convolution or pooling for downsampling?"Notes 3.2, 4.1

6.2Why CNNs, and the convolution

Interview 1Why are fully connected networks unsuitable for images?

Week 3Very likelyNotes 1.2From the lecture

Three problems.

Prints

inputs: 30000
first MLP layer: 30,001,000 parameters
32x32x3 -> 512 hidden: 1,573,376 (weights 1,572,864 + 512 biases)
conv layer, 32 filters of 3x3x3: 896 parameters (864 weights + 32 biases)

Prints (real vs scrambled pictures)

MLP  on the real pictures    : 1.0
MLP  on the scrambled pictures: 1.0
CNN  on the real pictures    : 1.0
CNN  on the scrambled pictures: 0.597

Answer

Say this first (30 seconds)

(1) Too many parameters: a 100x100 RGB image into 1000 hidden units is 30 million weights in layer 1, so slow training and overfitting. (2) Spatial structure is lost: flattening discards which pixels are neighbours; an MLP is exactly as accurate on pixel-scrambled images, proving it never used layout. (3) No translation invariance: a cat at a new position is a new problem; in the experiment an MLP trained on the left side scored 35% (chance) on the right.

Go deeper

A CNN addresses all three with locality and weight sharing.

Follow-up they may askWhat would data augmentation do for the MLP?

Interview 2What are locality and weight sharing, and why do they make CNNs work?

Week 3Very likelyNotes 1.3From the lecture

Two assumptions about images.

Prints

MLP: 16,643 parameters   test on LEFT 0.93   test on RIGHT (never seen there) 0.35
CNN:  1,299 parameters   test on LEFT 1.00   test on RIGHT (never seen there) 1.00
chance level is 0.33

Answer

Say this first (30 seconds)

Locality: what matters about a pixel is its neighbourhood, so each unit looks at a small window. Weight sharing: a pattern (an edge) means the same anywhere, so the same filter weights are reused at every position. Result: far fewer parameters, and a pattern learned once is detected everywhere (translation equivariance).

Go deeper

3x3 filter = 9 weights (per channel pair) against input_size x output_size for a dense layer.

Follow-up they may askEquivariance versus invariance?

Interview 3Explain the convolution operation in a CNN.

Week 3Very likelyNotes 2.1From the lecture

Slide, multiply, add.

Prints

by hand:
 tensor([[ 0., 27., 27.],
        [ 0., 27., 27.],
        [ 0., 27., 27.]])
window at (0,1) = [[0.0, 0.0, 9.0], [0.0, 0.0, 9.0], [0.0, 0.0, 9.0]] -> sum of products = 27.0
F.conv2d agrees: True

Answer

Say this first (30 seconds)

A small learnable matrix, the kernel, is placed on each window of the image; the window and kernel are multiplied element-wise and summed into one number; sliding over all positions gives a feature map showing where the pattern the kernel detects occurs. The kernel's weights are shared across all positions and learned by backpropagation.

Go deeper

Worked example: an edge image with a vertical-edge kernel gives 0 in the flat region and 27 where the window contains the edge.

Follow-up they may askIs nn.Conv2d a true convolution?

Interview 4How does a convolution filter work on an RGB image, and how many feature maps come out?

Week 3LikelyNotes 2.3From the lecture

A filter is a 3-D block.

Prints

weight shape: (32, 3, 3, 3)  = (out_channels, in_channels, kH, kW)
bias shape  : (32,)
parameters  : 896  = 3*3*3*32 + 32
input  : (8, 3, 32, 32)
output : (8, 32, 30, 30) <- 32 feature maps, each 30x30 (no padding)
Conv2d(  1,  8,k=3) parameters =      80  (= 3*3*1*8 + 8)

Answer

Say this first (30 seconds)

A filter has one 3x3 slice per input channel (3x3x3 for RGB). At each position the three slice-products are summed into one number, so one filter produces one feature map; $K$ filters produce $K$ maps = the next layer's $K$ channels. Weights tensor shape: $(C_{out},C_{in},K_h,K_w)$ .

Go deeper

Conv2d(3, 32, 3) has weight shape (32, 3, 3, 3) and 896 parameters.

Follow-up they may askWhat does a Conv2d(64, 128, 3) cost?

Interview 5Is the "convolution" in deep learning a real mathematical convolution?

Week 3LikelyNotes 2.2From the lecture

One flip.

Prints

PyTorch 'convolution' (no flip): [-2.0, -2.0]
true convolution (kernel flipped): [2.0, 2.0]
(they differ: the sign flips; for a LEARNED kernel it makes no difference)

Answer

Say this first (30 seconds)

Strictly no: mathematical convolution flips the kernel before sliding it; nn.Conv2d does not flip, so it computes a cross-correlation. It is harmless because the kernel is learned: if the flipped one works better, training finds it.

Go deeper

It matters only when you use a fixed, hand-made asymmetric kernel.

Follow-up they may askWhich classical kernel detects vertical edges?

Interview 6Is a convolution translation-invariant or translation-equivariant?

Week 3LikelyNotes 1.3Extra

Move the input; what moves?

Prints

MLP: 16,643 parameters   test on LEFT 0.93   test on RIGHT (never seen there) 0.35
CNN:  1,299 parameters   test on LEFT 1.00   test on RIGHT (never seen there) 1.00
chance level is 0.33

Answer

Say this first (30 seconds)

Equivariant: shifting the input shifts the feature map by the same amount. Invariant means the final output does not change; a CNN gets that by adding pooling and a global summary at the end (or a global max pool before the classifier), and only approximately.

Go deeper

In the experiment the global max pool turned equivariant feature maps into a position-independent answer: 100% on a never-seen region.

Follow-up they may askWhy is a flatten-then-linear head not invariant?

Interview 7What does a Sobel kernel do, and how is it related to CNN filters?

Week 3SometimesNotes 2.1Extra

A fixed edge detector.

Prints

. . . . . . . .
. . # # # # . .
. # # # # # # .
. # # . . # # .
. # # . . # # .
. # # # # # # .
. . # # # # . .
. . . . . . . .
pixels flagged as edge: 28 (the response hugs the border of the square, inside and out; flat regions give 0)

Answer

Say this first (30 seconds)

Sobel is a hand-designed 3x3 kernel that responds to intensity changes in one direction ( $[-1,0,1;-2,0,2;-1,0,1]$ for vertical edges); combining the x and y responses outlines objects. A CNN learns such kernels itself: in the bar experiment the first layer learned line and edge detectors without being told.

Go deeper

Classic computer vision hand-crafted features; deep learning learns them.

Follow-up they may askWhat is a "feature map"?

6.3Sizes, strides and padding

Interview 8A 32x32x3 input goes through Conv2d(3, 32, 5). What are the output shape and the parameter count?

Week 3Very likelyNotes 3.3, 2.3From the lecture

Use the formula, and remember the biases.

main.py

# input 32x32x3, Conv2d(3, 32, kernel_size=5), stride 1, padding 0
H_out = (32 + 2*0 - 5) // 1 + 1      # = 28
params = 5*5*3*32 + 32               # = 2432

Prints

combinations checked: 240  mismatches: 0
H=32 K=5 P=0 S=1  ->  (H+2P-K)/S+1 = 28  ->  floor 28
H=28 K=3 P=1 S=1  ->  (H+2P-K)/S+1 = 28  ->  floor 28

Answer

Say this first (30 seconds)

Output: $\lfloor(32+0-5)/1\rfloor+1=28$ , so (32, 28, 28) (32 feature maps). Parameters: $5\cdot5\cdot3\cdot32+32=\mathbf{2432}$ (2400 weights + 32 biases). The size does not depend on the image resolution, only on kernel and channels.

Go deeper

Forgetting the biases or the floor are the two usual slips.
Verified against PyTorch on 240 combinations of H, K, P, S.

Follow-up they may askWhat if the stride is 2?

Interview 9What is padding, and what do "valid" and "same" mean?

Week 3Very likelyNotes 3.2-3.3From the lecture

Zeros around the border.

Prints

K=3: padding P=(K-1)/2=1  output 28x28  (same as input: True)
K=5: padding P=(K-1)/2=2  output 28x28  (same as input: True)
K=7: padding P=(K-1)/2=3  output 28x28  (same as input: True)
padding='same' does it for you: (1, 4, 28, 28)
padding=0 ('valid'), K=5: (1, 4, 24, 24)  <- each conv shrinks the image

Answer

Say this first (30 seconds)

Padding adds a border of zeros before convolving. Valid = none: each conv shrinks the output (28x28 with a 5x5 kernel gives 24x24). Same = enough padding that, at stride 1, the output has the input's size: $P=(K-1)/2$ . It stops the feature map shrinking layer by layer and lets border pixels be used as often as central ones.

Go deeper

Odd kernels (3, 5, 7) pad symmetrically.

Follow-up they may askWhat padding keeps a 7x7 kernel size-preserving?

Interview 10What is a 1x1 convolution and what is it for?

Week 3LikelyNotes 3.4From the lecture

It cannot see neighbours.

Prints

1x1 conv: (2, 64, 10, 10) -> (2, 16, 10, 10)  height and width untouched
parameters: 1040  = 64*16 + 16
identical to a Linear layer at each pixel: True
3x3, 256->256 directly: 590,080   via 1x1 bottleneck (256->64->64->256): 70,016

Answer

Say this first (30 seconds)

A 1x1 kernel mixes the channels at each pixel (a per-pixel linear layer) without looking at neighbours. Uses: reduce or expand the number of channels cheaply, add non-linearity, and build bottlenecks (256 to 64 channels, a 3x3 there, back to 256: 70,016 weights instead of 590,080). The "network-in-network" idea behind Inception and ResNet.

Go deeper

Parameters $=C_{in}C_{out}+C_{out}$ ; verified equal to a Linear layer at every pixel.

Follow-up they may askWhere will you meet it next week?

Interview 11Stride-2 convolution or pooling to downsample?

Week 3SometimesNotes 3.2, 4.1Extra

Learned versus fixed.

Prints (stride 1 vs stride 2)

H=32 K=5 P=0 S=1  ->  (H+2P-K)/S+1 = 28  ->  floor 28
H=28 K=3 P=1 S=1  ->  (H+2P-K)/S+1 = 28  ->  floor 28
H=28 K=3 P=0 S=2  ->  (H+2P-K)/S+1 = 13.5  ->  floor 13
H=32 K=3 P=0 S=2  ->  (H+2P-K)/S+1 = 15.5  ->  floor 15
H= 7 K=3 P=0 S=2  ->  (H+2P-K)/S+1 = 3  ->  floor 3

Answer

Say this first (30 seconds)

Both halve the size. Pooling is a fixed summary (no parameters); a stride-2 convolution is a learned downsampling (has weights and non-linearity) and is used in many modern networks (all-convolutional nets, ResNet shortcuts). Pooling is simpler and gives a little shift tolerance.

Go deeper

Rule of thumb in exams: pooling has 0 parameters, a strided conv does not.

Follow-up they may askWhat is global average pooling?

6.4Pooling, receptive fields and shapes

Interview 12What does pooling do, and does it have parameters?

Week 3Very likelyNotes 4.1From the lecture

Summarise each window.

Prints

max pool:
 tensor([[6., 2.],
        [2., 9.]])
avg pool:
 tensor([[3.5000, 1.0000],
        [1.0000, 6.0000]])
parameters in a pooling layer: 0
shape: (1, 1, 4, 4) -> (1, 1, 2, 2) (height and width halved)

Prints

inputs equal: False
after 2x2 max pool, equal: True  <- small shifts inside a window disappear
shift across a window border, equal: False  <- pooling is only approximately invariant

Answer

Say this first (30 seconds)

Pooling downsamples: each window (2x2, stride 2) is replaced by its max or average, per channel. It has no learnable parameters. It cuts computation, grows the receptive field of later layers and gives some tolerance to small shifts. Max pooling keeps the strongest response ("the feature is in this window").

Go deeper

It is only approximately shift-invariant: a shift across a window border changes the output.

Follow-up they may askCan a stride-2 convolution replace pooling?

Interview 13Why use two 3x3 convolutions instead of one 5x5?

Week 3Very likelyNotes 4.2From the lecture

Same receptive field, fewer parameters, more non-linearity.

Prints

one 3x3 conv           : [3]
two 3x3 convs          : [3, 5]  <- equals one 5x5
three 3x3 convs        : [3, 5, 7]  <- equals one 7x7
conv3, pool2, conv3    : [3, 4, 8]
conv3, pool2 x3 blocks : [3, 4, 8, 10, 18, 22]

parameters with 64 channels in and out (no bias):
  three 3x3 convs: 110592    one 7x7 conv: 200704    (same 7x7 receptive field)
  two 3x3 convs  : 73728    one 5x5 conv: 102400    (same 5x5 receptive field)

Answer

Say this first (30 seconds)

Two stacked 3x3 layers have the same 5x5 receptive field but fewer weights ( $18C^2$ against $25C^2$ ), and a ReLU between them makes the pair more expressive. Three 3x3 equal a 7x7 for $27C^2$ against $49C^2$ . This is VGGNet's core idea.

Go deeper

With 64 channels: 73,728 against 102,400 weights, and 110,592 against 200,704.

Follow-up they may askWhat is a receptive field?

Interview 14Why does the first Linear layer after the convolutions give a shape error, and how do you fix it?

Week 3LikelyNotes 5.2Extra

Count the values after Flatten.

Prints

input                  (1, 1, 28, 28)
Conv2d                 (1, 8, 28, 28)     params 80
ReLU                   (1, 8, 28, 28)     params 0
MaxPool2d              (1, 8, 14, 14)     params 0
Conv2d                 (1, 16, 14, 14)    params 1168
ReLU                   (1, 16, 14, 14)    params 0
MaxPool2d              (1, 16, 7, 7)      params 0
Flatten                (1, 784)           params 0
Linear                 (1, 10)            params 7850
total parameters: 9098

Answer

Say this first (30 seconds)

Its in_features must equal $C\times H\times W$ of the last feature map. Changing the image size, stride, padding or pooling changes that number. Compute it with the output-size formula, or push a dummy tensor through the feature extractor and read the shape. (For 28x28 with two conv+pool blocks and 16 channels: $16\cdot7\cdot7=784$ .)

Go deeper

nn.AdaptiveAvgPool2d(1) removes the problem: it always outputs 1x1.

Follow-up they may askWhy do most of a small CNN's parameters sit in the last layer?

Next · SummaryWeek 3 Summary Sheet

Summary

Week 3 Summary Sheet

Week 3 on one page: chapter glance, key terms, pairs to tell apart, a formula sheet with worked numbers, the error messages and traps, a checklist and notes on the source.

7.1Week 3 at a glance

C1Why CNNs

MLP on images: 30M params in one layer, layout lost (scramble test), no position independence. Fix: locality + weight sharing.

C2Convolution

Slide a kernel, multiply element-wise, add. One filter = one feature map. Params $=K^2C_{in}C_{out}+C_{out}$ .

C3K, S, P

$H_{out}=\lfloor(H+2P-K)/S\rfloor+1$ . Same: $P=(K-1)/2$ . 1x1 conv mixes channels (bottleneck).

C4Pooling and RF

Pooling: 0 params, halves H, W. Receptive field $r\leftarrow r+(k-1)j$ , $j\leftarrow js$ . Three 3x3 = one 7x7 for 45% fewer weights.

C5Building a CNN

[Conv, ReLU, Pool] x n, Flatten, Linear. Trace shapes. Measured: MLP 35% on unseen positions, CNN 100% with 13x fewer params.

7.2Key terms

C1-C2

channel: one grid of an image / one feature map.
kernel / filter: the small learnable matrix.
feature map: where a pattern occurs.
weight sharing: same kernel at every position.

C3-C4

stride: step between windows.
padding: zeros around the border.
pooling: window summary, no params.
receptive field: input region a unit sees.

7.3The pairs you must tell apart

MLPflat, layout lost

CNNlocal, shared weights

validno padding, shrinks

same

P=(K-1)/2

, keeps size

equivariantoutput shifts with input

invariantoutput unchanged

convolutionflips the kernel (maths)

cross-correlationwhat Conv2d computes

3x3 convsees neighbours

1x1 convmixes channels only

max poolstrongest response

avg poolsmooth summary

poolingfixed, 0 params

strided convlearned downsampling

7.4Formula sheet with worked numbers

Quantity	Formula	Worked example (verified)
Output size	$\lfloor(H+2P-K)/S\rfloor+1$	32, K=5, P=0, S=1: 28 \| 28, K=3, P=0, S=2: floor(13.5) = 13
Same padding	$P=(K-1)/2$	K=3: 1, K=5: 2, K=7: 3
Conv parameters	$K_hK_wC_{in}C_{out}+C_{out}$	Conv2d(3,32,3): 864 + 32 = 896 \| Conv2d(64,128,3): 73,856
1x1 conv parameters	$C_{in}C_{out}+C_{out}$	Conv2d(64,64,1): 4,160
Dense layer for comparison	$n_{in}n_{out}+n_{out}$	3072 -> 512: 1,573,376
Receptive field	$r\leftarrow r+(k-1)j,\; j\leftarrow js$	3x3, 3x3: 5 \| conv3, pool2, conv3: 8
Pooling parameters	0	MaxPool2d(2): 0

7.5Errors and traps

You see	Cause	Fix
`mat1 and mat2 shapes cannot be multiplied` after the conv layers	wrong `in_features` of the first Linear	trace shapes with a dummy tensor; use AdaptiveAvgPool
`Calculated padded input size per channel ... Kernel size can't be greater than actual input size`	kernel larger than the (padded) input	add padding or shrink the kernel
output size is not an integer in the notes' formula	forgot the floor	PyTorch rounds down
accuracy collapses when objects move	MLP / flatten head, no shift tolerance	conv + global pooling, augmentation

7.6Last-minute checklist

Numbers

floor in the output size
+ biases in the parameter count
params ignore image size
pooling = 0 params

Ideas

locality + weight sharing
three 3x3 = one 7x7, fewer weights
1x1 = channel mixing
Conv2d = cross-correlation

7.7Corrections and additions to the source notes

Where	The source says	Better
Output size	$\frac{H+2P-K}{S}+1$	add the floor: $\lfloor\cdot\rfloor$ (13.5 becomes 13; verified on 240 combinations)
Parameter comparison	"3x3 twice: 9+9 = 18 vs 5x5: 25"	per channel pair; with C channels it is 18C^2 vs 25C^2 (28% fewer at C=64)
Translation invariance	"convolution gives translation invariance"	strictly equivariance; invariance comes from pooling / global summary
Examples	32x32x3 MLP vs conv	1,573,376 (with bias) vs 896: both verified by running
MLP vs CNN	claims only	measured: scrambled-pixel test (MLP unchanged, CNN 0.60) and position test (MLP 35%, CNN 100%)

Revision overview

WhyMLP: too many params
layout lost
no shift tolerance

Convolutionkernel slides
weight sharing
one filter, one map

K, S, Poutput size
same padding
1x1 conv

Week 3Convolutional networks

Pooling0 params
halves H, W
approx. invariance

Receptive fieldgrows with depth
3x3 stacks
VGG idea

BuildConv ReLU Pool
Flatten Linear
trace shapes

Revision: all of Week 3 on one page

Deep Learning,Week 3

Chapters

Images and Why MLPs Fail

1.1An image is a grid of numbers

What we ask computers to do with images

1.2Why an MLP is the wrong tool

Problem 1: far too many parameters

Problem 2: the spatial structure is thrown away

Problem 3: no translation invariance

1.3The two ideas behind CNNs

1.4Chapter summary

The Convolution Operation

2.1Slide, multiply, add

2.2Convolution or cross-correlation?

2.3Many filters, many channels

2.4What the layers learn: a hierarchy

2.5Chapter summary

Kernel Size, Stride, Padding

3.1Kernel size

3.2Stride and padding

3.3The output size formula

3.4The 1x1 convolution

3.5Chapter summary

Pooling and the Receptive Field

4.1Pooling

4.2The receptive field

4.3Chapter summary

Building a CNN

5.1The standard recipe

5.2A common mistake: the Flatten size

5.3Where the CNN wins: when the object moves

5.4Chapter summary

Interview Question Bank

6.1Interview heat-map: what is most likely

6.2Why CNNs, and the convolution

Go deeper

Go deeper

Go deeper

Go deeper

Go deeper

Go deeper

Go deeper

6.3Sizes, strides and padding

Go deeper

Go deeper

Go deeper

Go deeper

6.4Pooling, receptive fields and shapes

Go deeper

Go deeper

Go deeper

Week 3 Summary Sheet

7.1Week 3 at a glance

7.2Key terms

7.3The pairs you must tell apart

7.4Formula sheet with worked numbers

7.5Errors and traps

7.6Last-minute checklist

7.7Corrections and additions to the source notes

Deep Learning,
Week 3