Week 4 Notes

Deep Learning,
Week 4

Introduction to DL and GenAI, Week 4: batch normalisation, overfitting, dropout and weight decay, the classic architectures (AlexNet, VGG, GoogLeNet), ResNet and skip connections, transfer learning, and U-Net segmentation with IoU and Dice. Every number was really run, including the parameter counts of the real architectures.

8 chapters16 worked examples~44 min read

BatchNormDropoutAlexNet / VGGGoogLeNetResNetTransfer learningU-NetInterview prep

Chapters

01Batch NormalisationPart 1
02Overfitting, Dropout and Weight DecayPart 2
03Classic ArchitecturesPart 3
04ResNetPart 4
05Transfer LearningPart 5
06Segmentation and U-NetPart 6
07Interview Question BankInterview
08Week 4 Summary SheetSummary

Part 1

Batch Normalisation

Deep networks are hard to train: Week 1 showed gradients vanishing, and even with ReLU the scale of activations drifts from layer to layer. Batch normalisation (BN) is the layer that largely fixed this, and it is in almost every CNN built since 2015.

1.1The problem and the idea

While a network trains, the weights of early layers keep changing, so the distribution of inputs reaching a later layer keeps shifting underneath it. The original paper called this internal covariate shift: each layer chases a moving target. The fix is to re-centre and re-scale the inputs of a layer for every mini-batch.

Honest footnote. Later research (2018) found that BN's benefit is not mainly about covariate shift: it smooths the loss surface, so gradients are more predictable and larger learning rates are safe. The practical effects the notes list (faster training, higher learning rates, a little regularisation) are real; the "covariate shift" story is only one explanation of why.

1.2The algorithm

Figure 1. Four values normalised to mean 0 and standard deviation 1

\mu_B=\frac1m\sum_{i=1}^m x_i,\qquad \sigma_B^2=\frac1m\sum_{i=1}^m (x_i-\mu_B)^2,\qquad \hat x_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}},\qquad y_i=\gamma\,\hat x_i+\beta

For each feature (each channel, in a conv layer), BN computes the mean and variance over the mini-batch (and over height and width for images), normalises, then applies a learnable scale $\gamma$ and shift $\beta$ . The last step matters: it lets the network undo the normalisation if that is what works best (setting $\gamma=\sigma,\beta=\mu$ restores the original), so BN removes no expressive power. $\epsilon$ (about $10^{-5}$ ) avoids dividing by zero. The notes' example [2, 4, 6, 8]:

main.py

import torch, torch.nn as nn
x = torch.tensor([2., 4., 6., 8.])              # one feature, a mini-batch of four values
mu = x.mean()
var = ((x - mu) ** 2).mean()                    # the batch (biased) variance, divide by m
xhat = (x - mu) / torch.sqrt(var + 1e-5)
print("mean =", mu.item(), " variance =", var.item(), " std =", round(var.sqrt().item(), 4))
print("normalised:", [round(v, 2) for v in xhat.tolist()])

gamma, beta = 2.0, 1.0                           # the learnable scale and shift
print("with gamma=2, beta=1:", [round(v, 2) for v in (gamma * xhat + beta).tolist()])

bn = nn.BatchNorm1d(1)                           # the library layer, gamma=1, beta=0 at the start
print("nn.BatchNorm1d      :", [round(v, 2) for v in bn(x.reshape(4, 1)).flatten().tolist()])

Output

mean = 5.0  variance = 5.0  std = 2.2361
normalised: [-1.34, -0.45, 0.45, 1.34]
with gamma=2, beta=1: [-1.68, 0.11, 1.89, 3.68]
nn.BatchNorm1d      : [-1.34, -0.45, 0.45, 1.34]

Mean 5; variance $(9+1+1+9)/4=5$ , std $\sqrt5=2.236$ ; the normalised values are $-3/2.236=-1.34,\ -0.45,\ 0.45,\ 1.34$ , matching the notes and the library layer. Note that the variance divides by $m$ , not $m-1$ .

1.3Training mode versus evaluation mode

The batch statistics cannot be used at prediction time: a single test image has no "batch", and the answer for one image must not depend on its neighbours. So BN keeps running averages of the mean and variance during training (two buffers, updated but not learned) and, in eval() mode, uses those. Per channel it has 2 learned parameters ( $\gamma,\beta$ ) and 2 buffers:

main.py

import torch, torch.nn as nn
torch.manual_seed(0)
bn = nn.BatchNorm1d(2, momentum=0.5)
print("parameters (learned)   :", [(n, tuple(p.shape)) for n, p in bn.named_parameters()])
print("buffers (not learned)  :", [(n, tuple(b.shape)) for n, b in bn.named_buffers() if n != "num_batches_tracked"])

batch = torch.randn(32, 2) * 3 + 10             # features with mean ~10, std ~3
bn.train()
out = bn(batch)
print("train mode, per-feature mean/std of the output:", out.mean(0).abs().max().item() < 1e-5, out.std(0, unbiased=False).tolist())
print("running_mean after one batch:", [round(v, 2) for v in bn.running_mean.tolist()])

for _ in range(30):                              # more batches: the running statistics settle
    bn(torch.randn(32, 2) * 3 + 10)
print("running_mean after 30 more  :", [round(v, 2) for v in bn.running_mean.tolist()], " (true mean 10)")
print("running_var  after 30 more  :", [round(v, 2) for v in bn.running_var.tolist()], " (true var 9)")

bn.eval()                                        # eval: use the stored statistics, not the batch's
single = torch.tensor([[10., 10.]])
print("eval mode on a single sample:", [round(v, 2) for v in bn(single).flatten().tolist()], "(works: no batch needed)")
bn.train()
try:
    bn(single)                                   # train mode needs more than one value per channel
except ValueError as e:
    print("train mode, batch of 1: ValueError:", str(e)[:60])

Output

parameters (learned)   : [('weight', (2,)), ('bias', (2,))]
buffers (not learned)  : [('running_mean', (2,)), ('running_var', (2,))]
train mode, per-feature mean/std of the output: True [0.9999992251396179, 0.9999997615814209]
running_mean after one batch: [5.19, 5.08]
running_mean after 30 more  : [9.9, 9.81]  (true mean 10)
running_var  after 30 more  : [7.84, 8.88]  (true var 9)
eval mode on a single sample: [0.03, 0.06] (works: no batch needed)
train mode, batch of 1: ValueError: Expected more than 1 value per channel when training, got in

Features with true mean 10 and variance 9 give running averages that settle near 10 and 9 (here 9.9 and 7.8 to 8.9 after 31 batches).
In eval() a single sample works; in train mode a batch of one raises an error because there is nothing to average over.
Forgetting model.eval() at test time is the classic BN bug: the test accuracy swings with the batch composition.

1.4What BN buys you, measured

The experiment: a 10-layer sigmoid network (the Week 1 recipe for vanishing gradients), same optimiser, same learning rate, with and without BN:

main.py

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
X = torch.randn(512, 10)
y = ((X[:, 0] * X[:, 1] + X[:, 2] ** 2 - X[:, 3]) > 0.5).float()      # a non-linear target

def make(bn, depth=10, width=32):
    layers, d = [], 10
    for _ in range(depth):
        layers += [nn.Linear(d, width)] + ([nn.BatchNorm1d(width)] if bn else []) + [nn.Sigmoid()]
        d = width
    return nn.Sequential(*layers, nn.Linear(width, 1))

for bn in (False, True):
    torch.manual_seed(1)
    model = make(bn)
    opt = torch.optim.SGD(model.parameters(), lr=0.1)                 # same optimiser, same learning rate
    log = []
    for epoch in range(300):
        opt.zero_grad()
        loss = F.binary_cross_entropy_with_logits(model(X).squeeze(1), y)
        loss.backward(); opt.step()
        if epoch in (0, 49, 149, 299):
            log.append(round(loss.item(), 3))
    print(("with BN   " if bn else "without BN"), "loss at epochs 1, 50, 150, 300:", log)

Output

without BN loss at epochs 1, 50, 150, 300: [0.685, 0.683, 0.683, 0.683]
with BN    loss at epochs 1, 50, 150, 300: [0.675, 0.494, 0.308, 0.28]

Without BN the loss is stuck at 0.683 (barely better than a coin flip, $\ln2=0.693$ ): the signal does not get through ten sigmoids. With BN it falls steadily to 0.28. Same architecture, same learning rate; only the normalisation layers differ.

Effect	Why
faster training	stable activation scale, smoother loss
higher learning rates possible	less sensitive to the weight scale
mild regularisation	each sample is normalised with a slightly random batch: a bit of noise
less careful initialisation needed	layers re-normalise whatever arrives

Where it goes, and its limit

Standard CNN order: Conv, BN, ReLU (with the conv bias dropped, since BN's $\beta$ replaces it). BN depends on a decent batch size; with tiny batches its statistics are noisy. That is why sequence models and Transformers (Week 10) use Layer Normalisation, which normalises across the features of one sample.

1.5Chapter summary

BN: normalise per feature over the batch, then scale $\gamma$ and shift $\beta$ (learned). Params $2C$ per layer, plus 2 buffers.
Train: batch statistics. Eval: running averages. Always call eval().
Measured: a 10-sigmoid net stuck at 0.683 trains to 0.28 with BN.

Next · Part 2Overfitting, Dropout and Weight Decay

Part 2

Overfitting, Dropout and Weight Decay

A network with millions of parameters and a small dataset can memorise the training set perfectly while learning nothing that transfers. This chapter measures that failure and the tools against it: dropout, weight decay, and (for images) data augmentation.

2.1Overfitting, seen in the loss curves

The set-up: 80 training examples, 30 features of which only 3 carry signal, noisy labels, and a network with 256 hidden units twice (about 75,000 weights). It can fit 80 points, noise included. Track the loss on the training data and on 2000 fresh test examples:

main.py

import torch, torch.nn as nn, torch.nn.functional as F
n_feat, n_signal = 30, 3

def data(n, g):                                     # only 3 of the 30 features carry signal, and the labels are noisy
    X = torch.randn(n, n_feat, generator=g)
    y = (X[:, :n_signal].sum(1) + 0.8 * torch.randn(n, generator=g) > 0).float()
    return X, y

def run(seed, dropout=0.0, wd=0.0, epochs=200):
    g = torch.Generator().manual_seed(seed)
    Xtr, ytr = data(80, g); Xte, yte = data(2000, g)          # only 80 training examples
    torch.manual_seed(seed)
    m = nn.Sequential(nn.Linear(n_feat, 256), nn.ReLU(), nn.Dropout(dropout), nn.Linear(256, 256), nn.ReLU(), nn.Dropout(dropout), nn.Linear(256, 1))
    opt = torch.optim.AdamW(m.parameters(), lr=1e-3, weight_decay=wd)
    out = {}
    for e in range(1, epochs + 1):
        m.train(); opt.zero_grad()
        F.binary_cross_entropy_with_logits(m(Xtr).squeeze(1), ytr).backward(); opt.step()
        if e in (20, 50, 100, 200):
            m.eval()
            with torch.no_grad():
                bce = lambda X, y: F.binary_cross_entropy_with_logits(m(X).squeeze(1), y).item()
                out[e] = (bce(Xtr, ytr), bce(Xte, yte), ((m(Xte).squeeze(1) > 0).float() == yte).float().mean().item())
    return out

print("average of 5 runs; loss = binary cross-entropy (lower is better)")
for name, kw in [("no regularisation", {}), ("dropout 0.5", {"dropout": 0.5}), ("weight decay 1.0", {"wd": 1.0}), ("both", {"dropout": 0.5, "wd": 1.0})]:
    runs = [run(s, **kw) for s in range(5)]
    print(name)
    for e in (20, 50, 100, 200):
        tr, te, acc = (sum(r[e][i] for r in runs) / 5 for i in range(3))
        print(f"   epoch {e:3d}: train loss {tr:.3f}   test loss {te:.3f}   test accuracy {acc:.3f}")

Output

average of 5 runs; loss = binary cross-entropy (lower is better)
no regularisation
   epoch  20: train loss 0.132   test loss 0.543   test accuracy 0.738
   epoch  50: train loss 0.002   test loss 0.858   test accuracy 0.752
   epoch 100: train loss 0.000   test loss 0.980   test accuracy 0.753
   epoch 200: train loss 0.000   test loss 1.054   test accuracy 0.752
dropout 0.5
   epoch  20: train loss 0.348   test loss 0.554   test accuracy 0.730
   epoch  50: train loss 0.035   test loss 0.620   test accuracy 0.755
   epoch 100: train loss 0.001   test loss 0.885   test accuracy 0.760
   epoch 200: train loss 0.000   test loss 1.106   test accuracy 0.761
weight decay 1.0
   epoch  20: train loss 0.138   test loss 0.541   test accuracy 0.737
   epoch  50: train loss 0.002   test loss 0.818   test accuracy 0.752
   epoch 100: train loss 0.001   test loss 0.866   test accuracy 0.752
   epoch 200: train loss 0.001   test loss 0.866   test accuracy 0.750
both
   epoch  20: train loss 0.356   test loss 0.557   test accuracy 0.729
   epoch  50: train loss 0.040   test loss 0.604   test accuracy 0.755
   epoch 100: train loss 0.002   test loss 0.809   test accuracy 0.760
   epoch 200: train loss 0.001   test loss 0.894   test accuracy 0.762

Figure 2. The classic overfitting picture: the training loss collapses to zero while the test loss climbs

No regularisation: train loss goes to 0.000 by epoch 100, but test loss, best at epoch 20 (0.543), climbs to 1.054. The model is memorising the noise and becoming over-confident on wrong answers.
Weight decay (shrinking the weights towards zero each step) ends with test loss 0.866, and weight decay + dropout with 0.894, both below the 1.054 of doing nothing. Dropout alone shows its effect earlier: at epoch 50 the test loss is 0.620 against 0.858, but by epoch 200 it has caught up (1.106). Dropout slows overfitting; it does not stop it.
Test accuracy barely moves (0.75 to 0.76 in every run): regularisation mainly improves the calibration of the confidence, not the argmax, on this problem.

What to take from it. Do not expect dropout or weight decay to be magic. They help most when the model is large relative to the data and you train long; the dependable defence is early stopping (stop at the epoch of lowest validation loss, here about 20) plus more data. Reading the table honestly is part of the skill.

2.2Dropout

During training, each unit is switched off with probability $p$ on every step, so each step trains a different random sub-network. The network cannot rely on one neuron being present, so it must spread information across many (redundant, robust representations); the whole thing behaves like an average of many thinned networks. At evaluation time all units are on. To keep the expected activation the same, PyTorch scales the survivors by $1/(1-p)$ during training (inverted dropout, measured in Week 2); the notes describe the older equivalent of scaling by $1-p$ at inference. Either way the point is the same: the layer's expected output must not change between training and testing.

2.3Weight decay and augmentation

Weight decay penalises large weights (an L2 penalty), pulling the model towards simpler functions. In Adam use AdamW (Week 5), which applies the decay correctly.
Data augmentation creates new training images by label-preserving changes: random flips, small shifts and rotations, crops, colour jitter. It teaches the invariances you want (a flipped cat is still a cat) and is often the single most effective regulariser for images.
More data and early stopping are the other two defences.

2.4Chapter summary

Overfitting = train loss near 0, test loss rising. Read it from the curves.
Dropout (random sub-networks), weight decay (small weights), augmentation, early stopping, more data.
Measured: regularisation lowered the final test loss but barely moved accuracy here.

Next · Part 3Classic Architectures

Part 3

Classic Architectures

Between 2012 and 2015 four networks set the vocabulary of modern computer vision. Each solved a specific problem the previous one exposed, so read them as a story of problem, fix, not as a list to memorise.

Figure 3. Four landmark networks; the parameter counts are measured by building each model in torchvision

main.py

import torchvision.models as tvm
rows = [("AlexNet (2012)", tvm.alexnet(weights=None)), ("VGG-16", tvm.vgg16(weights=None)),
        ("GoogLeNet / Inception v1", tvm.googlenet(weights=None, aux_logits=False, init_weights=False)),
        ("ResNet-18", tvm.resnet18(weights=None)), ("ResNet-34", tvm.resnet34(weights=None)), ("ResNet-50", tvm.resnet50(weights=None))]
for name, m in rows:
    print(f"{name:26s} {sum(p.numel() for p in m.parameters()):>12,} parameters")

Output

AlexNet (2012)               61,100,840 parameters
VGG-16                      138,357,544 parameters
GoogLeNet / Inception v1      6,624,904 parameters
ResNet-18                    11,689,512 parameters
ResNet-34                    21,797,672 parameters
ResNet-50                    25,557,032 parameters

3.1AlexNet (2012): deep learning wins ImageNet

AlexNet won the ImageNet challenge by a huge margin and started the deep-learning era in vision. 5 convolutional + 3 fully connected layers, 61 million parameters. What it combined: ReLU activations (fast, no saturation), GPU training (two GPUs), dropout in the fully connected layers, and data augmentation.

Precision. The notes call AlexNet "the first to use ReLU". ReLU was in use before (2010), but AlexNet is what popularised it for deep CNNs, so say "popularised" in an answer.

3.2VGGNet (2014): depth through simplicity

VGG's rule: use only 3x3 convolutions (13 of them in VGG-16) and stack them deeper (16 or 19 weight layers). Two 3x3 layers see what one 5x5 sees with fewer weights (Week 3). The price is paid elsewhere:

main.py

import torchvision.models as tvm
m = tvm.vgg16(weights=None)
conv = sum(p.numel() for p in m.features.parameters())
fc = sum(p.numel() for p in m.classifier.parameters())
print(f"VGG-16 convolutional part: {conv:>12,}  ({conv / (conv + fc):.0%})")
print(f"VGG-16 fully connected   : {fc:>12,}  ({fc / (conv + fc):.0%})")
first_fc = m.classifier[0]
print("first FC layer:", first_fc.in_features, "->", first_fc.out_features, "=", f"{first_fc.in_features * first_fc.out_features:,}", "weights alone (25088 = 512 x 7 x 7)")
n3 = sum(1 for l in m.features if l.__class__.__name__ == "Conv2d" and l.kernel_size == (3, 3))
print("3x3 convolutions in VGG-16:", n3, " (all convs are 3x3)")

Output

VGG-16 convolutional part:   14,714,688  (11%)
VGG-16 fully connected   :  123,642,856  (89%)
first FC layer: 25088 -> 4096 = 102,760,448 weights alone (25088 = 512 x 7 x 7)
3x3 convolutions in VGG-16: 13  (all convs are 3x3)

89% of VGG-16's 138 million parameters are in its fully connected head; the first FC layer alone, 25088 inputs (the 512x7x7 feature map flattened) times 4096, has 103 million. The convolutions, which do the real feature extraction, need only 14.7 million. This is why later networks dropped the big FC layers.

3.3GoogLeNet / Inception (2014): width and efficiency

Figure 4. One Inception module: four parallel branches concatenated

Which kernel size is best, 1x1, 3x3 or 5x5? Use all of them in parallel and concatenate: the module sees the input at several scales and lets training decide. To keep it affordable, each expensive branch first uses a 1x1 convolution to shrink the channels (Week 3's bottleneck). The module below is the real first block of GoogLeNet:

main.py

import torch, torch.nn as nn
class Inception(nn.Module):
    """Four parallel branches over the same input, concatenated along the channel axis."""
    def __init__(self, cin, c1, c3r, c3, c5r, c5, cp):
        super().__init__()
        self.b1 = nn.Conv2d(cin, c1, 1)                                                     # 1x1
        self.b2 = nn.Sequential(nn.Conv2d(cin, c3r, 1), nn.ReLU(), nn.Conv2d(c3r, c3, 3, padding=1))   # 1x1 reduce, then 3x3
        self.b3 = nn.Sequential(nn.Conv2d(cin, c5r, 1), nn.ReLU(), nn.Conv2d(c5r, c5, 5, padding=2))   # 1x1 reduce, then 5x5
        self.b4 = nn.Sequential(nn.MaxPool2d(3, stride=1, padding=1), nn.Conv2d(cin, cp, 1))           # pool, then 1x1
    def forward(self, x):
        return torch.cat([self.b1(x), self.b2(x), self.b3(x), self.b4(x)], dim=1)

count = lambda m: sum(p.numel() for p in m.parameters())
block = Inception(192, 64, 96, 128, 16, 32, 32)         # the first Inception block of GoogLeNet (3a)
x = torch.zeros(1, 192, 28, 28)
print("output shape:", tuple(block(x).shape), " = 64 + 128 + 32 + 32 =", 64 + 128 + 32 + 32, "channels")
print("parameters with 1x1 reductions:", f"{count(block):,}")

naive = nn.ModuleList([nn.Conv2d(192, 64, 1), nn.Conv2d(192, 128, 3, padding=1), nn.Conv2d(192, 32, 5, padding=2)])
print("same branches without the reductions:", f"{count(naive) + 192 * 32 + 32:,}", " <- the 1x1 reductions save a lot")

Output

output shape: (1, 256, 28, 28)  = 64 + 128 + 32 + 32 = 256 channels
parameters with 1x1 reductions: 163,696
same branches without the reductions: 393,472  <- the 1x1 reductions save a lot

The 1x1 reductions cut the module from 393,472 to 163,696 parameters (2.4x). GoogLeNet also replaced the big FC head with global average pooling (each feature map reduced to one number), which is why the whole 22-layer network has only 6.6 million parameters, 21 times fewer than VGG-16, while being deeper and more accurate.

Network	Problem it solved	Key trick
AlexNet	make deep nets train and win	ReLU + GPU + dropout
VGG	how deep, how simple?	only 3x3 convs, 16-19 layers
GoogLeNet	which kernel size, and parameters?	Inception + 1x1 reduction + global avg pool
ResNet	deeper got worse	skip connections (next chapter)

3.4Chapter summary

AlexNet (61M): ReLU, GPUs, dropout. VGG-16 (138M): 3x3 only, 89% of parameters in the FC head. GoogLeNet (6.6M): Inception, 1x1 reductions, global average pooling.
Wider (parallel branches) and cheaper (1x1) beat bigger FC heads.

Next · Part 4ResNet

Part 4

ResNet

By 2015 everyone believed deeper was better, until an experiment showed it was not. ResNet explained why and fixed it with one line of code, and its idea, the skip connection, appears in almost every architecture since, including the Transformer.

4.1The degradation problem

Take a network that works, add more layers, and train: the accuracy gets worse. Importantly it is worse on the training data too, so this is not overfitting (overfitting means good train, bad test). It is an optimisation failure: a deeper network could at the very least copy the shallower one by making the extra layers compute the identity, but gradient descent cannot find that solution through a stack of non-linear layers.

4.2The fix: learn the residual

Figure 5. A residual block: the input is carried around the layers and added back

Instead of asking a block to learn the whole mapping $H(x)$ , let it learn only the residual $F(x)=H(x)-x$ and add the input back: $y=F(x)+x$ . Now doing nothing is easy: just push $F(x)$ towards zero (weights near zero), and the block becomes the identity. Learning a small change is easier than learning a whole function. When the shapes of $F(x)$ and $x$ differ (the block changes channels or downsamples), the shortcut gets a 1x1 convolution to match them:

main.py

import torch, torch.nn as nn
class BasicBlock(nn.Module):
    """ResNet-18/34 block: y = relu( F(x) + shortcut(x) )."""
    def __init__(self, cin, cout, stride=1):
        super().__init__()
        self.f = nn.Sequential(nn.Conv2d(cin, cout, 3, stride, 1, bias=False), nn.BatchNorm2d(cout), nn.ReLU(),
                               nn.Conv2d(cout, cout, 3, 1, 1, bias=False), nn.BatchNorm2d(cout))
        # when the shape changes, the shortcut needs a 1x1 conv so that F(x) and x can be added
        self.shortcut = nn.Identity() if (stride == 1 and cin == cout) else nn.Sequential(nn.Conv2d(cin, cout, 1, stride, bias=False), nn.BatchNorm2d(cout))
    def forward(self, x):
        return torch.relu(self.f(x) + self.shortcut(x))

count = lambda m: sum(p.numel() for p in m.parameters())
same = BasicBlock(64, 64)
down = BasicBlock(64, 128, stride=2)
x = torch.zeros(1, 64, 56, 56)
print("same-shape block :", tuple(same(x).shape), "shortcut is", same.shortcut.__class__.__name__, f" params {count(same):,}")
print("down-sampling one:", tuple(down(x).shape), "shortcut is a 1x1 conv with stride 2", f" params {count(down):,}")

Output

same-shape block : (1, 64, 56, 56) shortcut is Identity  params 73,984
down-sampling one: (1, 128, 28, 28) shortcut is a 1x1 conv with stride 2  params 230,144

4.3Why gradients now flow

y=F(x)+x\quad\Longrightarrow\quad \frac{\partial y}{\partial x}=\frac{\partial F}{\partial x}+1

That +1 is a direct path for the gradient from the output to every earlier block, so it cannot be multiplied away to nothing, however deep. Measure it: 30 blocks of linear+tanh, plain versus residual (a single changed line), gradient norm at the first and last block:

main.py

import torch, torch.nn as nn
torch.manual_seed(0)
def first_vs_last(residual, depth=30, width=32):
    class Block(nn.Module):
        def __init__(self):
            super().__init__()
            self.f = nn.Sequential(nn.Linear(width, width), nn.Tanh())
        def forward(self, x):
            return x + self.f(x) if residual else self.f(x)      # the whole of ResNet in one line
    torch.manual_seed(0)
    net = nn.Sequential(*[Block() for _ in range(depth)], nn.Linear(width, 1))
    net(torch.randn(64, width)).pow(2).mean().backward()
    grads = [b.f[0].weight.grad.norm().item() for b in net[:-1]]
    return grads[0], grads[-1]

for residual in (False, True):
    first, last = first_vs_last(residual)
    print(("residual" if residual else "plain   "), f"30 blocks: gradient at block 1 = {first:.2e}, at block 30 = {last:.2e}")

Output

plain    30 blocks: gradient at block 1 = 5.89e-09, at block 30 = 1.29e-01
residual 30 blocks: gradient at block 1 = 1.62e+01, at block 30 = 1.63e+01

Plain: the first block's gradient is $6\times10^{-9}$ , nine orders of magnitude below the last block's. Residual: 16.2 against 16.3, essentially equal everywhere. This is why a residual network can be 100+ layers deep (ResNet-152) and still train, and why it fixes both the degradation problem and vanishing gradients.

4.4Basic block and bottleneck block

ResNet-18 and 34 use the basic block (two 3x3 convs). ResNet-50 and up use the bottleneck block (1x1 reduce, 3x3, 1x1 expand), the Inception trick again. At 256 channels the difference is enormous:

main.py

import torch.nn as nn
count = lambda m: sum(p.numel() for p in m.parameters())
basic = nn.Sequential(nn.Conv2d(256, 256, 3, padding=1, bias=False), nn.Conv2d(256, 256, 3, padding=1, bias=False))
bottleneck = nn.Sequential(nn.Conv2d(256, 64, 1, bias=False), nn.Conv2d(64, 64, 3, padding=1, bias=False), nn.Conv2d(64, 256, 1, bias=False))
print("two 3x3 convs at 256 channels      :", f"{count(basic):,}")
print("bottleneck 1x1 -> 3x3 -> 1x1 (64)  :", f"{count(bottleneck):,}", f" ({count(basic) / count(bottleneck):.1f}x fewer)")
print("ResNet-50 uses the bottleneck block; ResNet-18/34 use the basic block.")

Output

two 3x3 convs at 256 channels      : 1,179,648
bottleneck 1x1 -> 3x3 -> 1x1 (64)  : 69,632  (16.9x fewer)
ResNet-50 uses the bottleneck block; ResNet-18/34 use the basic block.

Say this in the exam

"ResNet adds an identity skip connection, so each block learns a residual $F(x)=H(x)-x$ . The shortcut gives the gradient a direct path ( $\partial y/\partial x=\partial F/\partial x+1$ ), which fixes vanishing gradients, and makes the identity easy to represent, which fixes the degradation problem: deeper plain networks had higher training error." Mention it is added, not concatenated.

4.5Chapter summary

Degradation: deeper plain nets have worse training error: an optimisation failure, not overfitting.
Residual block $y=F(x)+x$ : identity is easy; gradient has a "+1" path (measured 16.2 vs 6e-9 at the first block of 30).
1x1 conv shortcut when shapes change; bottleneck block is 16.9x cheaper at 256 channels.

Next · Part 5Transfer Learning

Part 5

Transfer Learning

Training a CNN from scratch needs millions of images and days of compute. Transfer learning starts from a network already trained on a huge dataset (ImageNet: 1.2 million photos, 1000 classes) and adapts it to your small problem. It is how most real image projects, and later most language projects, are built.

5.1Why it works

Figure 6. Early layers hold generic features; only the last stage and the head are specific to the old task

Recall the hierarchy from Week 3: early layers learn edges and textures, which every image task needs; later layers learn task-specific parts and objects. So the early layers of an ImageNet model are useful for X-rays, satellite photos or product pictures, and only the last layers must be replaced and retrained.

5.2Two strategies

1. Feature extractor

Freeze the convolutional base (no weight changes).
Train only a new classifier head.
Use when: small dataset, task similar to the original.

2. Fine-tuning

Unfreeze some or all layers and keep training.
Use a very small learning rate so the useful pre-trained weights are nudged, not destroyed.
Use when: larger dataset or a different domain.

The mechanics in PyTorch take three lines: freeze every parameter, replace the head with a new layer (new layers are trainable by default), and give the optimiser only what is trainable. Here on ResNet-18 (built without downloading weights; in a real project you would load the pre-trained ones):

main.py

import torch.nn as nn
import torchvision.models as tvm

model = tvm.resnet18(weights=None)               # in practice: weights=tvm.ResNet18_Weights.DEFAULT (needs a download)
total = sum(p.numel() for p in model.parameters())

for p in model.parameters():                     # 1. freeze everything: the feature extractor stays as it is
    p.requires_grad = False
model.fc = nn.Linear(model.fc.in_features, 5)    # 2. replace the head with a NEW layer for 5 classes (trainable by default)

trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"total parameters : {total:,}")
print(f"trainable now    : {trainable:,}  ({trainable / total:.2%}: just the new 512x5 + 5 head)")
print("optimiser gets only:", [n for n, p in model.named_parameters() if p.requires_grad])

Output

total parameters : 11,689,512
trainable now    : 2,565  (0.02%: just the new 512x5 + 5 head)
optimiser gets only: ['fc.weight', 'fc.bias']

Only 2,565 of 11.7 million parameters (0.02%) get trained: the new 512x5 head and its biases. That is why this works with few images and a laptop CPU.

5.3Seeing the benefit: pre-train on one task, transfer to another

ImageNet weights cannot be downloaded offline, so here is a miniature version of the same story that can be run. Task A (lots of data): classify lines by orientation (horizontal, vertical, two diagonals). Task B (almost no data): classify plus, X and L shapes, with only 3 training images per class. Three models on task B, ten random trials each: a CNN from scratch, the task-A feature extractor frozen with a new head, and the same extractor fine-tuned:

main.py

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
S = 16
def draw(kind, n, noise=0.15):
    X = torch.zeros(n, 1, S, S)
    for i in range(n):
        r = torch.randint(4, 12, (1,)).item(); c = torch.randint(4, 12, (1,)).item()
        if kind == 0:   X[i, 0, r, c - 3:c + 4] = 1                                   # horizontal line
        elif kind == 1: X[i, 0, r - 3:r + 4, c] = 1                                   # vertical line
        elif kind == 2:
            for d in range(-3, 4): X[i, 0, r + d, c + d] = 1                          # diagonal \
        elif kind == 3:
            for d in range(-3, 4): X[i, 0, r + d, c - d] = 1                          # diagonal /
        elif kind == 4: X[i, 0, r, c - 3:c + 4] = 1; X[i, 0, r - 3:r + 4, c] = 1      # plus
        elif kind == 5:
            for d in range(-3, 4): X[i, 0, r + d, c + d] = 1; X[i, 0, r + d, c - d] = 1   # X shape
        else: X[i, 0, r - 3:r + 4, c] = 1; X[i, 0, r + 3, c:c + 4] = 1                # L shape
    return X + noise * torch.randn_like(X)

def dataset(kinds, n):
    X = torch.cat([draw(k, n) for k in kinds]); y = torch.cat([torch.full((n,), i) for i in range(len(kinds))])
    p = torch.randperm(len(X)); return X[p], y[p]

def features():
    return nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2), nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.AdaptiveMaxPool2d(1), nn.Flatten())

def fit(model, params, X, y, epochs=60, lr=1e-2):
    opt = torch.optim.Adam(params, lr)
    for _ in range(epochs):
        opt.zero_grad(); F.cross_entropy(model(X), y).backward(); opt.step()

def acc(model, X, y):
    model.eval()
    with torch.no_grad():
        return (model(X).argmax(1) == y).float().mean().item()

# Task A (plentiful data): four line orientations. Pre-train the feature extractor here.
XA, yA = dataset([0, 1, 2, 3], 300)
base = nn.Sequential(features(), nn.Linear(32, 4)); fit(base, base.parameters(), XA, yA)
print("task A (pre-training) accuracy:", acc(base, XA, yA))

# Task B (very little data): plus / X / L with ONLY 3 training images per class.
scores = {"from scratch": [], "frozen features + new head": [], "fine-tune everything": []}
for trial in range(10):
    torch.manual_seed(100 + trial)
    XB, yB = dataset([4, 5, 6], 3); XBt, yBt = dataset([4, 5, 6], 100)
    scratch = nn.Sequential(features(), nn.Linear(32, 3)); fit(scratch, scratch.parameters(), XB, yB)
    frozen = nn.Sequential(features(), nn.Linear(32, 3)); frozen[0].load_state_dict(base[0].state_dict())
    for p in frozen[0].parameters(): p.requires_grad = False
    fit(frozen, frozen[1].parameters(), XB, yB)
    tuned = nn.Sequential(features(), nn.Linear(32, 3)); tuned[0].load_state_dict(base[0].state_dict())
    fit(tuned, tuned.parameters(), XB, yB, lr=3e-3)
    for k, m in zip(scores, (scratch, frozen, tuned)): scores[k].append(acc(m, XBt, yBt))
for k, v in scores.items():
    print(f"{k:28s} mean test accuracy over 10 trials: {sum(v) / len(v):.3f}   worst trial: {min(v):.2f}")

Output

task A (pre-training) accuracy: 1.0
from scratch                 mean test accuracy over 10 trials: 0.981   worst trial: 0.88
frozen features + new head   mean test accuracy over 10 trials: 0.996   worst trial: 0.98
fine-tune everything         mean test accuracy over 10 trials: 0.997   worst trial: 0.98

Transfer gives a higher mean (99.6% and 99.7% against 98.1%) and, more importantly, a much better worst case: 0.98 against 0.88. With so little data, a model that starts from scratch depends on luck; one that starts with useful features does not. (Task B is easy enough that scratch is already decent; on harder problems the gap is larger.)

5.4Which strategy when?

Your data	Similar to the pre-training data	Different from it
Small	Freeze everything, train the head	Train only the top few layers (low-level features still transfer)
Large	Fine-tune all layers, small learning rate	Fine-tune a lot, or train from scratch

Note on the notes. The notes' last cell says "train from scratch". Even then, initialising from pre-trained weights usually converges faster than random initialisation, so fine-tuning is the common choice unless the domain is truly alien (and you have lots of data).

5.5Chapter summary

Reuse a network pre-trained on a big dataset; early layers are generic.
Feature extractor: freeze, new head (2,565 trainable of 11.7M). Fine-tune: unfreeze, tiny learning rate.
Measured: with 3 images per class transfer lifted the worst case from 0.88 to 0.98.

Next · Part 6Segmentation and U-Net

Part 6

Segmentation and U-Net

Classification gives one label per image. Semantic segmentation gives a label to every pixel (road, car, tumour, background). That needs a network whose output is as large as its input, and the standard design for it is the U-Net (originally for biomedical images).

6.1The U shape

Figure 7. An encoder that shrinks, a decoder that grows, and skip connections that carry detail across

Encoder (contracting path): conv + pooling repeatedly. Spatial size falls, channels rise: it learns what is in the image, but loses where precisely.
Decoder (expanding path): upsampling (a learned transposed convolution) + conv. Size grows back to the input's.
Skip connections: each encoder feature map is concatenated to the decoder map of the same size, handing back the fine spatial detail that pooling threw away.

main.py

import torch, torch.nn as nn
x = torch.zeros(1, 16, 8, 8)
up = nn.ConvTranspose2d(16, 8, kernel_size=2, stride=2)       # learned upsampling
y = up(x)
print("ConvTranspose2d:", tuple(x.shape), "->", tuple(y.shape), " (height and width doubled)")

skip = torch.zeros(1, 8, 16, 16)                               # the encoder feature map at the same resolution
cat = torch.cat([y, skip], dim=1)
print("concatenate with the skip:", tuple(cat.shape), " channels 8 + 8 =", cat.shape[1])
print("add instead (ResNet-style):", tuple((y + skip).shape), " <- channels stay 8, the two are summed together")

Output

ConvTranspose2d: (1, 16, 8, 8) -> (1, 8, 16, 16)  (height and width doubled)
concatenate with the skip: (1, 16, 16, 16)  channels 8 + 8 = 16
add instead (ResNet-style): (1, 8, 16, 16)  <- channels stay 8, the two are summed together

Concatenate, not add

ResNet adds the shortcut (shapes must match, the paths are summed). U-Net concatenates along the channel axis (8 + 8 = 16 channels): the decoder keeps both the upsampled coarse information and the encoder's detailed one, side by side, and lets the next convolution decide how to combine them. It is an exam favourite.

6.2Do the skips really matter? Measured

The task: two noisy rectangles per image, the answer is the exact pixel mask. A small U-Net (29,321 parameters), and the same network with the skip connections removed (the decoder gets only the upsampled map):

main.py

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
S = 32
def make(n):                                                   # noisy images containing two rectangles; the mask is the answer
    X = torch.zeros(n, 1, S, S); M = torch.zeros(n, 1, S, S)
    for i in range(n):
        for _ in range(2):
            h = torch.randint(5, 12, (1,)).item(); w = torch.randint(5, 12, (1,)).item()
            r = torch.randint(0, S - h, (1,)).item(); c = torch.randint(0, S - w, (1,)).item()
            X[i, 0, r:r + h, c:c + w] = 1; M[i, 0, r:r + h, c:c + w] = 1
    return X + 0.4 * torch.randn_like(X), M

def block(i, o):
    return nn.Sequential(nn.Conv2d(i, o, 3, padding=1), nn.ReLU(), nn.Conv2d(o, o, 3, padding=1), nn.ReLU())

class UNet(nn.Module):
    def __init__(self, skip=True):
        super().__init__(); self.skip = skip
        self.e1, self.e2, self.mid = block(1, 8), block(8, 16), block(16, 32)
        self.up2, self.d2 = nn.ConvTranspose2d(32, 16, 2, stride=2), block(32 if skip else 16, 16)
        self.up1, self.d1 = nn.ConvTranspose2d(16, 8, 2, stride=2), block(16 if skip else 8, 8)
        self.out, self.pool = nn.Conv2d(8, 1, 1), nn.MaxPool2d(2)
    def forward(self, x):
        e1 = self.e1(x); e2 = self.e2(self.pool(e1)); m = self.mid(self.pool(e2))      # encoder: shrink, deepen
        u2 = self.up2(m); d2 = self.d2(torch.cat([u2, e2], 1) if self.skip else u2)    # decoder: grow, with skips
        u1 = self.up1(d2); d1 = self.d1(torch.cat([u1, e1], 1) if self.skip else u1)
        return self.out(d1)

Xtr, Mtr = make(300); Xte, Mte = make(100)
def iou(logit, M):
    p = (logit > 0).float(); tp = (p * M).sum()
    return (tp / (p.sum() + M.sum() - tp)).item()

for skip in (True, False):
    torch.manual_seed(1)
    m = UNet(skip); opt = torch.optim.Adam(m.parameters(), 0.01)
    for _ in range(40):
        perm = torch.randperm(300)
        for i in range(0, 300, 30):
            idx = perm[i:i + 30]
            opt.zero_grad(); F.binary_cross_entropy_with_logits(m(Xtr[idx]), Mtr[idx]).backward(); opt.step()
    m.eval()
    with torch.no_grad():
        print(("U-Net with skip connections" if skip else "same network, NO skips        "), f"{sum(p.numel() for p in m.parameters()):,} params   test IoU {iou(m(Xte), Mte):.3f}")

Output

U-Net with skip connections 29,321 params   test IoU 0.987
same network, NO skips         26,441 params   test IoU 0.818

IoU 0.987 with skips against 0.818 without: the without-skip network knows roughly where the rectangles are but draws blurry, misplaced edges, because the detail was destroyed on the way down and nothing hands it back. Same data, same epochs; the 10% extra parameters are not what made the difference.

6.3Metrics: IoU and Dice

Pixel accuracy is misleading for segmentation: if the object covers 5% of the image, "everything is background" scores 95%. Two overlap measures between the predicted mask and the true mask:

\mathrm{IoU}=\frac{|A\cap B|}{|A\cup B|}=\frac{TP}{TP+FP+FN},\qquad \mathrm{Dice}=\frac{2|A\cap B|}{|A|+|B|}=\frac{2TP}{2TP+FP+FN},\qquad \mathrm{Dice}=\frac{2\,\mathrm{IoU}}{1+\mathrm{IoU}}

main.py

import torch
pred = torch.zeros(6, 6); pred[1:4, 1:4] = 1               # predicted mask: a 3x3 square
true = torch.zeros(6, 6); true[2:5, 2:5] = 1               # true mask: the same square shifted by one
tp = (pred * true).sum().item()
fp = (pred * (1 - true)).sum().item()                       # predicted but not real
fn = ((1 - pred) * true).sum().item()                       # real but missed
print(f"TP={tp:.0f}  FP={fp:.0f}  FN={fn:.0f}")
iou = tp / (tp + fp + fn)
dice = 2 * tp / (2 * tp + fp + fn)
print(f"IoU  = TP/(TP+FP+FN)      = {iou:.4f}")
print(f"Dice = 2TP/(2TP+FP+FN)    = {dice:.4f}")
print(f"Dice = 2*IoU/(1+IoU)      = {2 * iou / (1 + iou):.4f}   <- the two are linked")
print("pixel accuracy:", ((pred == true).float().mean().item()), " <- looks great because most pixels are background")

Output

TP=4  FP=5  FN=5
IoU  = TP/(TP+FP+FN)      = 0.2857
Dice = 2TP/(2TP+FP+FN)    = 0.4444
Dice = 2*IoU/(1+IoU)      = 0.4444   <- the two are linked
pixel accuracy: 0.7222222089767456  <- looks great because most pixels are background

Two 3x3 squares shifted by one pixel overlap in 4 pixels: IoU $=4/(4+5+5)=0.286$ , Dice $=8/(8+10)=0.444$ , pixel accuracy 0.72 (!). Dice is always at least as large as IoU, and the two rank models identically (one is a monotone function of the other); Dice is often used as a training loss (as $1-$ Dice) because it is differentiable when written with soft probabilities.

6.4Chapter summary

Segmentation = a label per pixel. U-Net: encoder (shrink), decoder (grow), skips (concatenate).
Measured: removing the skips dropped IoU from 0.987 to 0.818.
IoU = TP/(TP+FP+FN), Dice = 2TP/(2TP+FP+FN) = 2 IoU/(1+IoU); prefer them to pixel accuracy.

Next · InterviewInterview Question Bank

Interview

Interview Question Bank

What is likely to be asked from Week 4, with 30-second answers, deeper explanations, a lecture-or-extra tag on each question, and code you can run.

Week 4 questions are about why each trick exists: what BN stabilises, what skip connections fix, why 1x1 convolutions, why skips in a U-Net. Each answer starts with a 30-second version. Questions are tagged From the notes or Extra.

1Definition

One clear sentence: what it is.

2Intuition or tiny example

Why it works, in one picture or three lines.

3The failure it prevents

What goes wrong without it.

7.1Interview heat-map: what is most likely

Very likely8 topics

Batch normalisation"Explain batch normalisation. Training versus inference?"Notes 1.2-1.3

ResNet and skip connections"Why does ResNet work? What is the degradation problem?"Notes 4.1-4.3

Dropout"How does dropout work? Training versus inference?"Notes 2.2

Transfer learning"What is transfer learning? Feature extraction vs fine-tuning?"Notes 5.1-5.4

U-Net"Explain U-Net. Why skip connections, and why concatenate?"Notes 6.1

IoU and Dice"Define IoU and Dice. Why not pixel accuracy?"Notes 6.3

Overfitting and remedies"How do you recognise and fight overfitting?"Notes 2.1-2.3

Inception and 1x1 convolutions"What is an Inception module?"Notes 3.3

Likely5 topics

VGG and AlexNet facts"What are the key points of AlexNet and VGG?"Notes 3.1-3.2

BN parameters and placement"How many parameters does a BatchNorm layer have? Where does it go?"Notes 1.3-1.4

BN with small batches"What are BN's limitations?"Notes 1.4

Bottleneck block"Basic block versus bottleneck block in ResNet?"Notes 4.4

Global average pooling"What is global average pooling, and why replace FC layers with it?"Notes 3.3

Sometimes3 topics

Data augmentation"What is data augmentation? Name some transforms."Notes 2.3

Dice versus IoU"Which is larger, Dice or IoU?"Notes 6.3

Early stopping"What is early stopping?"Notes 2.1

7.2Normalisation and regularisation

Interview 1Explain batch normalisation, and how it differs between training and inference.

Week 4Very likelyNotes 1.2-1.3From the lecture

Normalise, then scale and shift.

Prints (the notes' example)

mean = 5.0  variance = 5.0  std = 2.2361
normalised: [-1.34, -0.45, 0.45, 1.34]
with gamma=2, beta=1: [-1.68, 0.11, 1.89, 3.68]
nn.BatchNorm1d      : [-1.34, -0.45, 0.45, 1.34]

Prints

without BN loss at epochs 1, 50, 150, 300: [0.685, 0.683, 0.683, 0.683]
with BN    loss at epochs 1, 50, 150, 300: [0.675, 0.494, 0.308, 0.28]

Answer

Say this first (30 seconds)

For each feature BN computes the mini-batch mean and variance, normalises $\hat x=(x-\mu)/\sqrt{\sigma^2+\epsilon}$ , then applies learnable scale $\gamma$ and shift $\beta$ . Effects: faster training, higher learning rates, mild regularisation. In training it uses batch statistics and updates running averages; at inference it uses the stored running mean and variance, so one sample can be predicted on its own.

Go deeper

Parameters $2C$ per layer ( $\gamma,\beta$ ); buffers: running mean and var.
Measured: a 10-sigmoid network stuck at loss 0.683 trained to 0.28 with BN.

Follow-up they may askWhy does BN fail with a batch of size 1 in training?

Interview 2How many learnable parameters does a BatchNorm layer have, and where is it placed?

Week 4LikelyNotes 1.3-1.4From the lecture

Per channel.

Prints

parameters (learned)   : [('weight', (2,)), ('bias', (2,))]
buffers (not learned)  : [('running_mean', (2,)), ('running_var', (2,))]
train mode, per-feature mean/std of the output: True [0.9999992251396179, 0.9999997615814209]

Answer

Say this first (30 seconds)

$2C$ learnable parameters ( $\gamma$ and $\beta$ per channel), plus 2C buffers (running mean and variance) that are updated but not learned by gradient descent. Usual placement: Conv, BN, ReLU, with the conv bias omitted because $\beta$ replaces it.

Go deeper

A common slip is to forget the buffers, or to count BN as having zero parameters like pooling.

Follow-up they may askWhere does BN go in a Transformer?

Interview 3What are the limitations of batch normalisation?

Week 4LikelyNotes 1.4Extra

It needs a batch.

Prints

eval mode on a single sample: [0.03, 0.06] (works: no batch needed)
train mode, batch of 1: ValueError: Expected more than 1 value per channel when training, got in

Answer

Say this first (30 seconds)

BN needs a reasonably large batch for stable statistics; with tiny batches (or a batch of 1 in training, which raises an error) it is noisy. It couples the samples in a batch, and has awkward behaviour for variable-length sequences. Alternatives that normalise within one sample: Layer Norm (Transformers, RNNs), Group Norm, Instance Norm.

Go deeper

Measured: train mode on a batch of one raises ValueError: Expected more than 1 value per channel.

Follow-up they may askWhich normalisation does a Transformer use?

Interview 4How does dropout work, and what changes at inference?

Week 4Very likelyNotes 2.2From the lecture

Random sub-networks.

Prints (excerpt)

average of 5 runs; loss = binary cross-entropy (lower is better)
no regularisation
   epoch  20: train loss 0.132   test loss 0.543   test accuracy 0.738
   epoch  50: train loss 0.002   test loss 0.858   test accuracy 0.752
   epoch 100: train loss 0.000   test loss 0.980   test accuracy 0.753
   epoch 200: train loss 0.000   test loss 1.054   test accuracy 0.752
dropout 0.5
   epoch  20: train loss 0.348   test loss 0.554   test accuracy 0.730
   epoch  50: train loss 0.035   test loss 0.620   test accuracy 0.755
   epoch 100: train loss 0.001   test loss 0.885   test accuracy 0.760

Answer

Say this first (30 seconds)

In training each unit is zeroed with probability $p$ , so every step trains a different thinned network; this prevents co-adaptation and forces redundant, robust features. At inference all units are on. The expected activation must match: PyTorch scales survivors by $1/(1-p)$ in training (inverted dropout); the older convention scales by $1-p$ at inference. Same effect.

Go deeper

In the overfitting experiment dropout slowed the rise of the test loss (0.620 against 0.858 at epoch 50) but did not stop it.

Follow-up they may askWhy must you call model.eval() before testing?

Interview 5How do you recognise overfitting, and what can you do about it?

Week 4Very likelyNotes 2.1-2.3Extra

Train loss down, test loss up.

Prints (excerpt)

average of 5 runs; loss = binary cross-entropy (lower is better)
no regularisation
   epoch  20: train loss 0.132   test loss 0.543   test accuracy 0.738
   epoch  50: train loss 0.002   test loss 0.858   test accuracy 0.752
   epoch 100: train loss 0.000   test loss 0.980   test accuracy 0.753
   epoch 200: train loss 0.000   test loss 1.054   test accuracy 0.752

Answer

Say this first (30 seconds)

The training loss keeps falling (here to 0.000) while the validation/test loss rises (0.543 at epoch 20 to 1.054 at epoch 200). Remedies: more data, data augmentation, dropout, weight decay, a smaller model, and early stopping (stop at the lowest validation loss). Always judge on held-out data.

Go deeper

Regularisation here lowered the final test loss but barely changed accuracy: read both metrics.

Follow-up they may askHow does overfitting differ from the degradation problem?

Interview 6What is early stopping?

Week 4SometimesNotes 2.1Extra

Stop when validation stops improving.

Prints (no regularisation)

no regularisation
   epoch  20: train loss 0.132   test loss 0.543   test accuracy 0.738
   epoch  50: train loss 0.002   test loss 0.858   test accuracy 0.752
   epoch 100: train loss 0.000   test loss 0.980   test accuracy 0.753
   epoch 200: train loss 0.000   test loss 1.054   test accuracy 0.752

Answer

Say this first (30 seconds)

Monitor the validation loss each epoch and stop (keeping the best weights) when it has not improved for some number of epochs (the "patience"). In the experiment the best test loss was at about epoch 20 (0.543); training to epoch 200 made it 1.054. It is free regularisation: you just train less.

Go deeper

It needs a validation set separate from the test set.

Follow-up they may askWhy not use the test set to decide when to stop?

Interview 7What is data augmentation, and what are some typical transforms?

Week 4SometimesNotes 2.3Extra

New images that keep the label.

main.py

tfm = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.RandomRotation(10),
    transforms.RandomResizedCrop(224, scale=(0.8, 1.0)),
    transforms.ColorJitter(0.2, 0.2),
    transforms.ToTensor(),
])   # applied freshly to each image every epoch

Answer

Say this first (30 seconds)

Creating extra training examples by label-preserving random changes: flips, small rotations and shifts, crops, scaling, colour jitter, noise. It teaches the invariances you want and acts as a strong regulariser. Choose transforms that keep the label true (do not flip a "6" into a "9").

Go deeper

Augmentation is applied to the training set only, never to validation or test.

Follow-up they may askWhy does an MLP gain less from translation augmentation than a CNN?

7.3Architectures

Interview 8What are the key ideas of AlexNet and VGG, and where are VGG's parameters?

Week 4LikelyNotes 3.1-3.2From the lecture

AlexNet: ReLU, GPU, dropout. VGG: 3x3 only.

Prints

AlexNet (2012)               61,100,840 parameters
VGG-16                      138,357,544 parameters
GoogLeNet / Inception v1      6,624,904 parameters
ResNet-18                    11,689,512 parameters
ResNet-34                    21,797,672 parameters
ResNet-50                    25,557,032 parameters

Prints

VGG-16 convolutional part:   14,714,688  (11%)
VGG-16 fully connected   :  123,642,856  (89%)
first FC layer: 25088 -> 4096 = 102,760,448 weights alone (25088 = 512 x 7 x 7)

Answer

Say this first (30 seconds)

AlexNet (2012, 5 conv + 3 FC, 61M parameters) won ImageNet by a wide margin using ReLU, GPUs, dropout and augmentation. VGG (16-19 layers) uses only 3x3 convolutions stacked deep: two 3x3 = one 5x5 field with fewer weights. In VGG-16, 89% of the 138M parameters are in the three fully connected layers (the first alone has 103M).

Go deeper

Remember the notes' wording: AlexNet popularised ReLU rather than inventing it.

Follow-up they may askWhich network in the list has the fewest parameters, and why?

Interview 9What is an Inception module, and what do the 1x1 convolutions do in it?

Week 4Very likelyNotes 3.3From the lecture

Parallel scales, cheap channels.

Prints

output shape: (1, 256, 28, 28)  = 64 + 128 + 32 + 32 = 256 channels
parameters with 1x1 reductions: 163,696
same branches without the reductions: 393,472  <- the 1x1 reductions save a lot

Answer

Say this first (30 seconds)

Four parallel branches (1x1, 3x3, 5x5 and pooling) look at the same input at several scales and are concatenated along channels, so the network chooses. Before the expensive 3x3 and 5x5 branches a 1x1 convolution shrinks the channels, cutting parameters (393,472 to 163,696 in the first block). GoogLeNet also uses global average pooling instead of big FC layers: 6.6M parameters in total.

Go deeper

Output channels = sum of the branch channels (64 + 128 + 32 + 32 = 256).

Follow-up they may askWhat does global average pooling do to the classifier head?

Interview 10What is the degradation problem and how does ResNet solve it?

Week 4Very likelyNotes 4.1-4.3From the lecture

Not overfitting. Say why.

Prints

plain    30 blocks: gradient at block 1 = 5.89e-09, at block 30 = 1.29e-01
residual 30 blocks: gradient at block 1 = 1.62e+01, at block 30 = 1.63e+01

Answer

Say this first (30 seconds)

Adding layers to a plain network made training error go up: not overfitting but an optimisation failure (the extra layers could just be the identity, but SGD cannot find that). ResNet adds an identity skip connection, $y=F(x)+x$ : the block learns the residual $F(x)=H(x)-x$ , so the identity is easy (push $F\to0$ ), and the gradient has a direct path, $\partial y/\partial x=\partial F/\partial x+1$ , so it does not vanish.

Go deeper

Measured over 30 blocks: first-block gradient 16.2 (residual) against 6e-9 (plain).
When the shapes differ the shortcut uses a 1x1 convolution (with stride) to match them.

Follow-up they may askResNet adds; what does U-Net do?

Interview 11What is the difference between the basic block and the bottleneck block in ResNet?

Week 4LikelyNotes 4.4Extra

How many channels does the 3x3 see?

Prints

two 3x3 convs at 256 channels      : 1,179,648
bottleneck 1x1 -> 3x3 -> 1x1 (64)  : 69,632  (16.9x fewer)
ResNet-50 uses the bottleneck block; ResNet-18/34 use the basic block.

Answer

Say this first (30 seconds)

Basic (ResNet-18/34): two 3x3 convs. Bottleneck (ResNet-50 and above): 1x1 (reduce channels), 3x3 (at the reduced width), 1x1 (expand back). The 3x3 runs on, say, 64 channels instead of 256: at 256 channels the bottleneck needs 69,632 parameters against 1,179,648, 16.9 times fewer.

Go deeper

Both add a skip connection around the block.

Follow-up they may askWhich ResNet is deeper per parameter?

Interview 12What is global average pooling and why do modern networks use it?

Week 4LikelyNotes 3.3Extra

Average each whole feature map.

main.py

x = torch.zeros(8, 512, 7, 7)               # a batch of 8 feature maps
gap = nn.AdaptiveAvgPool2d(1)
print(gap(x).flatten(1).shape)          # torch.Size([8, 512])
# an FC head on the flattened map would need 512*7*7*classes weights; GAP needs 0 and fixes the size

Answer

Say this first (30 seconds)

It averages each entire feature map down to one number (a $C\times H\times W$ map becomes $C$ numbers), so the classifier needs only $C\times$ classes weights. It has no parameters, works for any input size, and avoids the enormous FC layers of AlexNet and VGG (89% of VGG-16).

Go deeper

GoogLeNet and ResNet end with it.

Follow-up they may askHow does it differ from a flatten + linear head?

7.4Transfer learning and segmentation

Interview 13What is transfer learning, and when do you freeze versus fine-tune?

Week 4Very likelyNotes 5.1-5.4From the lecture

Early layers are generic.

Prints

total parameters : 11,689,512
trainable now    : 2,565  (0.02%: just the new 512x5 + 5 head)
optimiser gets only: ['fc.weight', 'fc.bias']

Prints

from scratch                 mean test accuracy over 10 trials: 0.981   worst trial: 0.88
frozen features + new head   mean test accuracy over 10 trials: 0.996   worst trial: 0.98
fine-tune everything         mean test accuracy over 10 trials: 0.997   worst trial: 0.98

Answer

Say this first (30 seconds)

Start from a network pre-trained on a big dataset (ImageNet) and adapt it. Early layers hold generic features (edges, textures), later ones are task-specific. Feature extractor: freeze the base, train a new head: for a small dataset similar to the original. Fine-tuning: unfreeze (some or all) layers with a very small learning rate: for a larger dataset or a different domain.

Go deeper

Replacing the head of ResNet-18 trains 2,565 of 11.7M parameters (0.02%).
Measured: with 3 images per class, transfer raised the worst-case accuracy from 0.88 to 0.98.

Follow-up they may askWhy a small learning rate when fine-tuning?

Interview 14Explain the U-Net. Why skip connections, and why concatenation rather than addition?

Week 4Very likelyNotes 6.1From the lecture

Encoder, decoder, and detail that pooling destroyed.

Prints

ConvTranspose2d: (1, 16, 8, 8) -> (1, 8, 16, 16)  (height and width doubled)
concatenate with the skip: (1, 16, 16, 16)  channels 8 + 8 = 16
add instead (ResNet-style): (1, 8, 16, 16)  <- channels stay 8, the two are summed together

Prints

U-Net with skip connections 29,321 params   test IoU 0.987
same network, NO skips         26,441 params   test IoU 0.818

Answer

Say this first (30 seconds)

An encoder (conv + pooling) shrinks the image and learns what; a decoder (upsampling + conv) grows it back to a per-pixel map. Pooling loses precise where information, so each encoder map is concatenated into the decoder map of equal size, handing back the fine detail. Concatenation keeps both sources separate (channels 8 + 8 = 16) and lets the next conv learn how to mix them; ResNet instead adds to enable identity and gradient flow.

Go deeper

Measured: skips on 0.987 IoU, skips off 0.818, with the same data and epochs.

Follow-up they may askWhat is a transposed convolution?

Interview 15Define IoU and the Dice coefficient. Why not use pixel accuracy for segmentation?

Week 4Very likelyNotes 6.3From the lecture

Overlap, not agreement.

Prints

TP=4  FP=5  FN=5
IoU  = TP/(TP+FP+FN)      = 0.2857
Dice = 2TP/(2TP+FP+FN)    = 0.4444
Dice = 2*IoU/(1+IoU)      = 0.4444   <- the two are linked
pixel accuracy: 0.7222222089767456  <- looks great because most pixels are background

Answer

Say this first (30 seconds)

$\mathrm{IoU}=TP/(TP+FP+FN)$ (intersection over union); $\mathrm{Dice}=2TP/(2TP+FP+FN)$ ; and $\mathrm{Dice}=2\,\mathrm{IoU}/(1+\mathrm{IoU})$ . Pixel accuracy counts the (many) easy background pixels, so a bad mask still scores high: in the example IoU was 0.29 and Dice 0.44 while pixel accuracy was 0.72.

Go deeper

Dice $\ge$ IoU always, and they rank models identically.

Follow-up they may askWhich would you use as a training loss, and why?

Interview 16For the same masks, which is larger, Dice or IoU? Are they equivalent?

Week 4SometimesNotes 6.3From the lecture

Use $\mathrm{Dice}=2\mathrm{IoU}/(1+\mathrm{IoU})$ .

Prints

IoU  = TP/(TP+FP+FN)      = 0.2857
Dice = 2TP/(2TP+FP+FN)    = 0.4444
Dice = 2*IoU/(1+IoU)      = 0.4444   <- the two are linked
pixel accuracy: 0.7222222089767456  <- looks great because most pixels are background

Answer

Say this first (30 seconds)

Dice $\ge$ IoU, with equality only at 0 and 1 (IoU 0.286 gives Dice 0.444). Because Dice is a monotone function of IoU they rank models the same way; they differ in how they weight partial overlaps, and Dice is the more common training loss.

Go deeper

Both ignore the (huge) true-negative background, unlike pixel accuracy.

Follow-up they may askWhat is the Dice of a perfect prediction?

Next · SummaryWeek 4 Summary Sheet

Summary

Week 4 Summary Sheet

Week 4 on one page: chapter glance, key terms, pairs to tell apart, a comparison table of the architectures with measured parameter counts, a decision guide for transfer learning, a checklist and notes on the source.

8.1Week 4 at a glance

C1BatchNorm

Normalise per feature over the batch, then $\gamma x+\beta$ . Train: batch stats; eval: running averages. Params $2C$ . Rescued a 10-sigmoid net (0.683 to 0.28).

C2Overfitting

Train loss to 0, test loss up. Dropout (random sub-nets), weight decay, augmentation, early stopping, more data.

C3Classic CNNs

AlexNet 61M (ReLU, GPU, dropout). VGG-16 138M (3x3 only; 89% in FC). GoogLeNet 6.6M (Inception, 1x1, global avg pool).

C4ResNet

$y=F(x)+x$ : learn the residual. Fixes degradation and vanishing gradients ( $\partial y/\partial x=\partial F/\partial x+1$ ). Bottleneck 16.9x cheaper.

C5Transfer learning

Reuse a pre-trained net. Small + similar data: freeze, new head. More data: fine-tune with a tiny learning rate.

C6U-Net

Encoder (shrink) + decoder (grow) + concatenated skips. IoU = TP/(TP+FP+FN), Dice = 2TP/(2TP+FP+FN). Skips took IoU from 0.818 to 0.987.

8.2Architectures at a glance (measured)

Network	Parameters	Idea
AlexNet (2012)	61,100,840	ReLU, GPU, dropout; 5 conv + 3 FC
VGG-16 (2014)	138,357,544	only 3x3; 14.7M conv + 123.6M FC
GoogLeNet (2014)	6,624,904	Inception, 1x1 reduce, global avg pool
ResNet-18	11,689,512	basic residual blocks
ResNet-34	21,797,672	basic residual blocks
ResNet-50	25,557,032	bottleneck residual blocks

8.3Key terms

C1-C2

BatchNorm: normalise per channel over the batch.
running mean/var: buffers used at eval.
dropout: random units off in training.
weight decay: L2 penalty on weights.

C3-C4

Inception: parallel branches, concatenated.
global avg pool: each map to one number.
skip connection: shortcut that adds the input back.
degradation: deeper plain net, worse training error.

C5-C6

feature extractor: frozen pre-trained base.
fine-tuning: unfrozen, tiny learning rate.
upsampling: transposed convolution in the decoder.
IoU / Dice: overlap measures of masks.

8.4The pairs you must tell apart

BN trainbatch statistics

BN evalrunning averages

BatchNormover the batch

LayerNormover one sample's features

overfittingtrain good, test bad

degradationtrain bad too

ResNet skipadds

U-Net skipconcatenates

feature extractorfrozen base, small data

fine-tuningunfrozen, small learning rate

IoUTP/(TP+FP+FN)

Dice2TP/(2TP+FP+FN)

basic blocktwo 3x3

bottleneck1x1, 3x3, 1x1

8.5Last-minute checklist

C1-C2

BN: $\gamma,\beta$ learned; eval uses running stats
always model.eval()
overfitting = rising test loss
early stopping at best validation loss

C3-C4

VGG: 3x3 only; AlexNet: ReLU, GPU, dropout
Inception: 1x1 reduce + concat
ResNet: $y=F(x)+x$ , +1 in the gradient
degradation is not overfitting

C5-C6

freeze small/similar, fine-tune larger
tiny learning rate when fine-tuning
U-Net concatenates
Dice = 2 IoU/(1+IoU)

8.6Corrections and additions to the source notes

Where	The source says	Better
AlexNet	"first to use ReLU"	it popularised ReLU for deep CNNs; ReLU existed earlier
Dropout	"inference: scale by $1-p$ "	PyTorch uses inverted dropout (scale by $1/(1-p)$ in training); equivalent
BatchNorm	"fixes internal covariate shift"	the benefit is real; later work attributes it to a smoother loss surface
VGG parameters	"only 3x3, fewer parameters"	VGG-16 has 138M, 89% in the FC layers: the saving is per receptive field, not overall
Transfer table	"large data + different domain: train from scratch"	fine-tuning from pre-trained weights usually still converges faster
Regularisation	"dropout/BN reduce overfitting"	measured: they slowed the rise of the test loss; accuracy barely changed

Revision overview

BatchNormnormalise per batch
train vs eval
2C params

Regularisedropout
weight decay
augmentation, early stop

Classic CNNsAlexNet, VGG
GoogLeNet
1x1 + global pool

Week 4Making CNNs deep and useful

ResNety = F(x) + x
degradation
bottleneck

Transferfreeze + head
fine-tune
small lr

U-Netencoder, decoder
concat skips
IoU, Dice

Revision: all of Week 4 on one page

Deep Learning,Week 4

Chapters

Batch Normalisation

1.1The problem and the idea

1.2The algorithm

1.3Training mode versus evaluation mode

1.4What BN buys you, measured

1.5Chapter summary

Overfitting, Dropout and Weight Decay

2.1Overfitting, seen in the loss curves

2.2Dropout

2.3Weight decay and augmentation

2.4Chapter summary

Classic Architectures

3.1AlexNet (2012): deep learning wins ImageNet

3.2VGGNet (2014): depth through simplicity

3.3GoogLeNet / Inception (2014): width and efficiency

3.4Chapter summary

ResNet

4.1The degradation problem

4.2The fix: learn the residual

4.3Why gradients now flow

4.4Basic block and bottleneck block

4.5Chapter summary

Transfer Learning

5.1Why it works

5.2Two strategies

5.3Seeing the benefit: pre-train on one task, transfer to another

5.4Which strategy when?

5.5Chapter summary

Segmentation and U-Net

6.1The U shape

6.2Do the skips really matter? Measured

6.3Metrics: IoU and Dice

6.4Chapter summary

Interview Question Bank

7.1Interview heat-map: what is most likely

7.2Normalisation and regularisation

Go deeper

Go deeper

Go deeper

Go deeper

Go deeper

Go deeper

Go deeper

7.3Architectures

Go deeper

Go deeper

Go deeper

Go deeper

Go deeper

7.4Transfer learning and segmentation

Go deeper

Go deeper

Go deeper

Go deeper

Week 4 Summary Sheet

8.1Week 4 at a glance

8.2Architectures at a glance (measured)

8.3Key terms

8.4The pairs you must tell apart

8.5Last-minute checklist

8.6Corrections and additions to the source notes

Deep Learning,
Week 4