Deep Learning,
Week 10

Week 9 ended with a gap: attention treats its input as a set, so "dog bites man" and "man bites dog" look the same. A Transformer must be told where each token sits. The fix is to add a vector that depends on the position to every token embedding. The original Transformer uses a fixed, parameter-free sinusoidal positional encoding.

1.1The formula

Each position $pos$ gets a vector of length $d$ (the model width). Dimensions come in sine-cosine pairs, and each pair oscillates at its own frequency: the first pair has wavelength $2\pi\approx6.3$ positions (a fast wave that changes at every token), the last has wavelength about $2\pi\cdot10000$ (a very slow wave that barely moves over thousands of tokens). Fast dimensions give fine-grained position, slow ones coarse position, like the hands of a clock reading seconds, minutes and hours. The encoding is added to the token embedding (not concatenated), so the model sees "word identity + position" together.

import numpy as np
def positional_encoding(n_pos, d):
    pos = np.arange(n_pos)[:, None]; i = np.arange(d // 2)[None, :]
    angle = pos / 10000 ** (2 * i / d)
    pe = np.zeros((n_pos, d)); pe[:, 0::2] = np.sin(angle); pe[:, 1::2] = np.cos(angle)     # PE(pos,2i)=sin, PE(pos,2i+1)=cos
    return pe

pe = positional_encoding(6, 8)
np.set_printoptions(precision=3, suppress=True, linewidth=120)
print("PE for positions 0..5, d = 8 (one row per position; columns come in (sin, cos) pairs, fastest frequency first):")
print(pe)
print("all values lie in [-1, 1]:", bool(np.abs(pe).all() <= 1))
big = positional_encoding(1000, 64)
dist = np.linalg.norm(big[:, None, :] - big[None, :, :], axis=-1)
print("1000 positions, d=64: smallest distance between two different positions =", round(float(dist[dist > 0].min()), 3), "(every position gets a distinct code)")
print("wavelengths of the (sin, cos) pairs for d=8:", [round(2 * np.pi * 10000 ** (2 * k / 8), 1) for k in range(4)], " (from 2*pi up to 2*pi*10000^(3/4))")

PE for positions 0..5, d = 8 (one row per position; columns come in (sin, cos) pairs, fastest frequency first):
[[ 0.     1.     0.     1.     0.     1.     0.     1.   ]
 [ 0.841  0.54   0.1    0.995  0.01   1.     0.001  1.   ]
 [ 0.909 -0.416  0.199  0.98   0.02   1.     0.002  1.   ]
 [ 0.141 -0.99   0.296  0.955  0.03   1.     0.003  1.   ]
 [-0.757 -0.654  0.389  0.921  0.04   0.999  0.004  1.   ]
 [-0.959  0.284  0.479  0.878  0.05   0.999  0.005  1.   ]]
all values lie in [-1, 1]: True
1000 positions, d=64: smallest distance between two different positions = 1.472 (every position gets a distinct code)
wavelengths of the (sin, cos) pairs for d=8: [6.3, 62.8, 628.3, 6283.2]  (from 2*pi up to 2*pi*10000^(3/4))

Position 0 is $[0,1,0,1,\dots]$ (sine of 0 is 0, cosine is 1). Reading down the first column pair, the values swing quickly (0, 0.84, 0.91, 0.14, $-0.76$, $-0.96$); in the last pair they barely move (0.001, 0.002, 0.003, ...). All values stay in $[-1,1]$, and among 1,000 positions at $d=64$ no two codes are closer than 1.47: every position has its own distinct signature. There is nothing to learn, and the encoding can be computed for positions longer than any seen in training.

1.2It encodes relative position too

Absolute positions are only half the story: language cares about "the word 3 places back". The sinusoidal design has a beautiful property. By the angle-addition identities, $\mathrm{PE}(pos+k)$ is a fixed linear function of $\mathrm{PE}(pos)$ for every $k$ (each sine-cosine pair is rotated by an angle that depends only on $k$). Test both consequences:

import numpy as np
def pe(pos, d=16):
    i = np.arange(d // 2)
    a = pos / 10000 ** (2 * i / d)
    out = np.zeros(d); out[0::2] = np.sin(a); out[1::2] = np.cos(a)
    return out

# 1. the dot product of two encodings depends only on the OFFSET between the positions
print("PE(p) . PE(p+k) for several starting positions p (columns) and offsets k (rows):")
print("  k \\ p     0       10      50      200")
for k in (0, 1, 2, 5, 10, 30, 100):
    print(f"  {k:3d}    " + "  ".join(f"{pe(p) @ pe(p + k):6.3f}" for p in (0, 10, 50, 200)))

# 2. PE(p + k) is a fixed LINEAR function of PE(p): find it by least squares from 200 positions and test on 50 unseen ones
k = 7
A = np.stack([pe(p) for p in range(200)]); B = np.stack([pe(p + k) for p in range(200)])
M = np.linalg.lstsq(A, B, rcond=None)[0]
test = np.stack([pe(p) for p in range(500, 550)]); true = np.stack([pe(p + k) for p in range(500, 550)])
print("\nshift by", k, ": one matrix maps PE(p) to PE(p+7) for EVERY p; max error on 50 unseen positions =", f"{np.abs(test @ M - true).max():.2e}")
print("so the model can learn 'look 7 tokens back' as a simple linear operation on the encodings")

PE(p) . PE(p+k) for several starting positions p (columns) and offsets k (rows):
  k \ p     0       10      50      200
    0     8.000   8.000   8.000   8.000
    1     7.485   7.485   7.485   7.485
    2     6.368   6.368   6.368   6.368
    5     6.137   6.137   6.137   6.137
   10     3.646   3.646   3.646   3.646
   30     2.699   2.699   2.699   2.699
  100     3.487   3.487   3.487   3.487

shift by 7 : one matrix maps PE(p) to PE(p+7) for EVERY p; max error on 50 unseen positions = 1.18e-08
so the model can learn 'look 7 tokens back' as a simple linear operation on the encodings

First, PE(p).PE(p+k) is identical for p = 0, 10, 50 and 200: it is a function of the offset $k$ alone (8.000 at $k=0$, which is $d/2$, falling to 7.485 at $k=1$, 6.368 at $k=2$, 3.646 at $k=10$). It is not a monotone distance, though: at $k=100$ it is back up to 3.487, because the slow waves line up again. Second, a single matrix maps $\mathrm{PE}(p)$ to $\mathrm{PE}(p+7)$ for every $p$, fitted on 200 positions and accurate to $10^{-8}$ on 50 unseen ones. So "look 7 tokens back" can be learned as a plain linear operation, which is what the attention projections can do.

1.3Chapter summary

• Attention is order-blind (Week 9), so add a position vector to each token embedding.
• $\mathrm{PE}(pos,2i)=\sin(pos/10000^{2i/d})$, $\mathrm{PE}(pos,2i+1)=\cos(\cdot)$: bounded, unique per position, parameter-free; low dimensions are fast waves, high dimensions slow.
• Measured: $\mathrm{PE}(p)\cdot\mathrm{PE}(p+k)$ depends only on $k$, and one matrix maps $\mathrm{PE}(p)$ to $\mathrm{PE}(p+7)$ to $10^{-8}$: relative position is built in.

Attention is only one ingredient. A Transformer layer wraps it with three more: a position-wise feed-forward network, residual connections and layer normalisation. Stacked $N$ times, this is the whole architecture of "Attention Is All You Need" (Vaswani et al., 2017).

2.1Layer normalisation

For one token with feature vector $x\in\mathbb{R}^H$, LayerNorm computes the mean and variance over the $H$ features of that token, $\mu=\tfrac1H\sum_ix_i$, $\sigma^2=\tfrac1H\sum_i(x_i-\mu)^2$, normalises $\hat x=(x-\mu)/\sqrt{\sigma^2+\epsilon}$, and applies a learned scale and shift $y=\gamma\hat x+\beta$. Compare with BatchNorm of Week 4, which uses the statistics of one feature over the batch:

import torch, torch.nn as nn
torch.manual_seed(0)
x = torch.randn(4, 5, 8) * 3 + 2                                 # a batch of 4 sequences, 5 tokens each, 8 features
ln = nn.LayerNorm(8); bn = nn.BatchNorm1d(8)
y_ln = ln(x)
by_hand = (x - x.mean(-1, keepdim=True)) / torch.sqrt(x.var(-1, unbiased=False, keepdim=True) + 1e-5)
print("LayerNorm equals (x - mean) / std taken over the FEATURES of each token:", torch.allclose(y_ln, by_hand, atol=1e-5))
print("every single token now has mean", round(y_ln[0, 0].mean().item(), 6), "and std", round(y_ln[0, 0].std(unbiased=False).item(), 4))

# does a token's output depend on the OTHER sequences in the batch?
x2 = x.clone(); x2[1:] = torch.randn(3, 5, 8) * 10                 # replace the other 3 sequences
print("\nchange the other sequences in the batch:")
print("  LayerNorm output of sequence 0 unchanged:", torch.allclose(ln(x)[0], ln(x2)[0]))
bn.train()
b1 = bn(x.reshape(-1, 8).clone()).reshape(4, 5, 8)[0]; bn2 = nn.BatchNorm1d(8).train()
b2 = bn2(x2.reshape(-1, 8).clone()).reshape(4, 5, 8)[0]
print("  BatchNorm output of sequence 0 unchanged:", torch.allclose(b1, b2), " (its statistics came from the whole batch)")
single = torch.randn(1, 1, 8)
print("\na batch of ONE token: LayerNorm works ->", ln(single).shape, "; BatchNorm in train mode:", end=" ")
try:
    nn.BatchNorm1d(8).train()(single.reshape(-1, 8)); print("works")
except ValueError as e:
    print("ValueError:", str(e)[:48])
print("parameters: LayerNorm has", sum(p.numel() for p in ln.parameters()), "(gamma and beta per feature) and needs no running statistics")

LayerNorm equals (x - mean) / std taken over the FEATURES of each token: True
every single token now has mean -0.0 and std 1.0

change the other sequences in the batch:
  LayerNorm output of sequence 0 unchanged: True
  BatchNorm output of sequence 0 unchanged: False  (its statistics came from the whole batch)

a batch of ONE token: LayerNorm works -> torch.Size([1, 1, 8]) ; BatchNorm in train mode: ValueError: Expected more than 1 value per channel when trai
parameters: LayerNorm has 16 (gamma and beta per feature) and needs no running statistics

The hand formula matches nn.LayerNorm, and every token comes out with mean 0 and standard deviation 1. The key test is the next block: replace the other sequences in the batch and LayerNorm's output for sequence 0 does not change, while BatchNorm's does (its statistics came from the whole batch). LayerNorm also works on a batch of one token, where BatchNorm raises an error, and it needs no running averages, so training and inference behave the same. That is why Transformers use it: sequences have variable length and padding, batches may be small, and each token should stand on its own.

2.2Residual connections and the feed-forward network

Every sub-layer is wrapped as $\mathrm{LayerNorm}(x+\mathrm{SubLayer}(x))$: the residual connection (Week 4) adds the input back, so a layer can start as a near-identity and gradients reach early layers directly, which is what makes 12, 24 or 96 layers trainable. The second sub-layer is a position-wise feed-forward network: the same two-layer MLP, $d\to4d\to d$ with a ReLU (or GELU) between, applied to each token independently. Attention mixes information between tokens; the FFN then processes each token on its own, and holds most of the layer's parameters.

2.3The encoder layer, written out

The notes' template, line for line: x = LayerNorm(x + MultiHeadSelfAttention(x)), then x = LayerNorm(x + FFN(x)). Reproduce PyTorch's layer from its own weights:

import torch, torch.nn as nn
torch.manual_seed(0)
d, heads, ff, n = 16, 4, 64, 5
layer = nn.TransformerEncoderLayer(d, heads, ff, dropout=0.0, batch_first=True, norm_first=False)     # the "Add & Norm after" layout of the original paper
layer.eval()
x = torch.randn(2, n, d)

attn_out = layer.self_attn(x, x, x, need_weights=False)[0]
h1 = layer.norm1(x + attn_out)                                    # 1. multi-head self-attention, then Add & Norm
ffn = layer.linear2(torch.relu(layer.linear1(h1)))                # 2. position-wise feed-forward: d -> 4d -> d
out = layer.norm2(h1 + ffn)                                       #    then Add & Norm
print("hand-written encoder layer matches nn.TransformerEncoderLayer:", torch.allclose(out, layer(x), atol=1e-5))
print("shapes: input", tuple(x.shape), " after attention", tuple(attn_out.shape), " FFN hidden", tuple(layer.linear1(h1).shape), " output", tuple(out.shape), "(same as the input: layers can be stacked)")

# the FFN works on each position independently: permute the tokens -> the FFN output is permuted the same way
perm = torch.randperm(n)
print("position-wise FFN: ffn(x[perm]) == ffn(x)[perm]:", torch.allclose(layer.linear2(torch.relu(layer.linear1(h1[:, perm]))), ffn[:, perm], atol=1e-6))

hand-written encoder layer matches nn.TransformerEncoderLayer: True
shapes: input (2, 5, 16)  after attention (2, 5, 16)  FFN hidden (2, 5, 64)  output (2, 5, 16) (same as the input: layers can be stacked)
position-wise FFN: ffn(x[perm]) == ffn(x)[perm]: True

The six lines reproduce nn.TransformerEncoderLayer exactly. The output has the same shape as the input, which is what lets layers be stacked, and the feed-forward really is position-wise (shuffling the tokens shuffles its output identically). Note that this is the post-norm layout of the original paper (normalise after adding). Most modern models use pre-norm (x + SubLayer(LayerNorm(x)), norm_first=True), which is commonly reported to train more stably at depth; the tiny GPT later in this week uses it.

2.4The decoder layer

The decoder layer has three sub-layers, each followed by Add & Norm: (1) masked self-attention over the target tokens produced so far (causal mask, Week 9); (2) cross-attention, with the queries from the decoder and the keys and values from the encoder output; (3) the feed-forward network. Check the two promises with code:

import torch, torch.nn as nn
torch.manual_seed(0)
d, heads, ff = 16, 4, 64
layer = nn.TransformerDecoderLayer(d, heads, ff, dropout=0.0, batch_first=True).eval()
memory = torch.randn(1, 6, d)                           # the encoder's output: 6 source tokens
tgt = torch.randn(1, 4, d)                              # 4 target tokens produced so far
causal = nn.Transformer.generate_square_subsequent_mask(4)
print("causal mask used by the decoder's FIRST attention (-inf = blocked):")
print(causal.numpy())
out = layer(tgt, memory, tgt_mask=causal)
print("\nsub-layers: 1) masked self-attention over the 4 target tokens (4x4), 2) cross-attention to the 6 encoder positions (4x6), 3) FFN")
print("output shape", tuple(out.shape), "= one vector per target token")

tgt2 = tgt.clone(); tgt2[0, 3] = torch.randn(d)         # change the LAST target token
out2 = layer(tgt2, memory, tgt_mask=causal)
print("changing the last target token leaves outputs 0, 1, 2 unchanged:", torch.allclose(out[:, :3], out2[:, :3], atol=1e-6), " (no leak from the future)")
memory2 = memory.clone(); memory2[0, 5] = torch.randn(d)
print("changing an ENCODER token changes every decoder output (cross-attention):", not torch.allclose(out, layer(tgt, memory2, tgt_mask=causal), atol=1e-6))

causal mask used by the decoder's FIRST attention (-inf = blocked):
[[  0. -inf -inf -inf]
 [  0.   0. -inf -inf]
 [  0.   0.   0. -inf]
 [  0.   0.   0.   0.]]

sub-layers: 1) masked self-attention over the 4 target tokens (4x4), 2) cross-attention to the 6 encoder positions (4x6), 3) FFN
output shape (1, 4, 16) = one vector per target token
changing the last target token leaves outputs 0, 1, 2 unchanged: True  (no leak from the future)
changing an ENCODER token changes every decoder output (cross-attention): True

The mask is lower-triangular (-inf above the diagonal, 0 elsewhere). Changing the last target token leaves the outputs for tokens 0, 1, 2 exactly unchanged, so nothing leaks from the future; yet changing one encoder token changes every decoder output, because every decoder position can attend to all encoder positions. In generation the decoder runs once per output token, appending the new token and feeding it back, exactly the seq2seq loop of Week 8, now with attention instead of a recurrent state.

2.5How many parameters? The notes get this wrong

The notes list the right pieces, self-attention $4d^2$ and FFN $8d^2$, then state "per layer about $8d^2$", which contradicts them. Count what PyTorch builds:

import torch.nn as nn
count = lambda m: sum(p.numel() for p in m.parameters())
d, ff = 512, 2048                                       # the original Transformer "base" sizes, feed-forward = 4d
enc = nn.TransformerEncoderLayer(d, 8, ff, batch_first=True)
dec = nn.TransformerDecoderLayer(d, 8, ff, batch_first=True)
attention = count(enc.self_attn); ffn = count(enc.linear1) + count(enc.linear2); norms = count(enc.norm1) + count(enc.norm2)
print(f"encoder layer, d = {d}:")
print(f"  self-attention   {attention:>10,}   = 4 d^2 + 4 d      = {4 * d * d + 4 * d:,}")
print(f"  feed-forward     {ffn:>10,}   = 8 d^2 + 5 d      = {8 * d * d + 5 * d:,}")
print(f"  layer norms      {norms:>10,}   = 4 d             = {4 * d:,}")
print(f"  TOTAL            {count(enc):>10,}   = {count(enc) / d ** 2:.3f} d^2  (the notes say ~8 d^2; the attention and FFN parts they list add to 4 + 8 = 12 d^2)")
print(f"decoder layer, d = {d}: {count(dec):,} = {count(dec) / d ** 2:.3f} d^2  (an extra cross-attention block: 4 + 4 + 8 = 16 d^2)")
print(f"original base model, 6 encoder + 6 decoder layers (no embeddings): {count(nn.Transformer(d, 8, 6, 6, ff)):,}")

encoder layer, d = 512:
  self-attention    1,050,624   = 4 d^2 + 4 d      = 1,050,624
  feed-forward      2,099,712   = 8 d^2 + 5 d      = 2,099,712
  layer norms           2,048   = 4 d             = 2,048
  TOTAL             3,152,384   = 12.025 d^2  (the notes say ~8 d^2; the attention and FFN parts they list add to 4 + 8 = 12 d^2)
decoder layer, d = 512: 4,204,032 = 16.037 d^2  (an extra cross-attention block: 4 + 4 + 8 = 16 d^2)
original base model, 6 encoder + 6 decoder layers (no embeddings): 44,140,544

An encoder layer has 3,152,384 parameters, $12.025\,d^2$ at $d=512$: attention $4d^2+4d$, feed-forward $8d^2+5d$ (the $8d^2$ is two matrices $d\times4d$ and $4d\times d$) and $4d$ for the two layer norms. So the rule of thumb is $12d^2$ per encoder layer, and a decoder layer with its extra cross-attention is about $16d^2$ (4,204,032). The original 6+6-layer "base" model has 44.1 million parameters before embeddings. The feed-forward network holds two thirds of each layer's weights, not attention.

2.6Chapter summary

• Each sub-layer is $\mathrm{LayerNorm}(x+\mathrm{SubLayer}(x))$. Encoder: self-attention, FFN. Decoder: masked self-attention, cross-attention, FFN.
• LayerNorm normalises over the features of one token (independent of the batch; verified), BatchNorm over the batch. FFN: $d\to4d\to d$ per position.
• Measured: encoder layer $=12.025\,d^2$ (3,152,384 at $d=512$), decoder layer $16.04\,d^2$; the notes' "$8d^2$ per layer" is wrong.

The Transformer of the original paper was an encoder plus a decoder, built for translation. Taking its two halves apart gave the model families that dominate AI today. The difference between them is almost entirely which attention mask is used and what the model is trained to predict.

3.1The three families

import torch
n = 6
bidirectional = torch.ones(n, n, dtype=torch.bool)                                   # BERT: every token sees every token
causal = torch.tril(torch.ones(n, n, dtype=torch.bool))                              # GPT: token t sees tokens 0..t
print("BERT (encoder-only): who can each token attend to?  1 = allowed")
print(bidirectional.int().numpy())
print("GPT (decoder-only, causal):")
print(causal.int().numpy())
print("tokens visible to each position, BERT:", bidirectional.sum(1).tolist(), "  GPT:", causal.sum(1).tolist())
print("BERT sees the right context too, which suits 'understand the sentence' tasks; GPT cannot, which is what allows it to generate left to right")

BERT (encoder-only): who can each token attend to?  1 = allowed
[[1 1 1 1 1 1]
 [1 1 1 1 1 1]
 [1 1 1 1 1 1]
 [1 1 1 1 1 1]
 [1 1 1 1 1 1]
 [1 1 1 1 1 1]]
GPT (decoder-only, causal):
[[1 0 0 0 0 0]
 [1 1 0 0 0 0]
 [1 1 1 0 0 0]
 [1 1 1 1 0 0]
 [1 1 1 1 1 0]
 [1 1 1 1 1 1]]
tokens visible to each position, BERT: [6, 6, 6, 6, 6, 6]   GPT: [1, 2, 3, 4, 5, 6]
BERT sees the right context too, which suits 'understand the sentence' tasks; GPT cannot, which is what allows it to generate left to right

The two masks make the difference concrete. In BERT every token sees all 6 positions; in GPT the token at position $t$ sees only $t$ of them (1, 2, 3, 4, 5, 6). The causal mask is what makes GPT a generator: nothing in its input can depend on a token that does not yet exist. BERT, which sees both sides, cannot be sampled left to right, but gives richer representations of a sentence that is already complete.

3.2How big do they get?

With a layer costing about $12d^2$ parameters, the size of a model is easy to estimate: $\text{layers}\times12d^2$ plus the embedding tables (vocabulary $\times\,d$ and positions $\times\,d$). Build the layers in PyTorch and add the embeddings:

import torch.nn as nn
count = lambda m: sum(p.numel() for p in m.parameters())
def stack(d, layers, heads):
    return count(nn.TransformerEncoder(nn.TransformerEncoderLayer(d, heads, 4 * d, batch_first=True), layers))
print("parameters = (layers) x (about 12 d^2) + embeddings")
for name, d, layers, heads, vocab, positions in (("BERT-base ", 768, 12, 12, 30522, 512), ("GPT-2 small", 768, 12, 12, 50257, 1024), ("BERT-large", 1024, 24, 16, 30522, 512)):
    body = stack(d, layers, heads); emb = vocab * d + positions * d
    print(f"{name}: layers {body / 1e6:7.1f}M  +  token & position embeddings {emb / 1e6:5.1f}M  =  {(body + emb) / 1e6:6.1f}M   (12 L d^2 = {12 * layers * d * d / 1e6:.1f}M)")
print("close to the published sizes (about 110M, 124M and 335-340M); the small remainder is the pooler, segment embeddings and layer norms we left out")

parameters = (layers) x (about 12 d^2) + embeddings
BERT-base : layers    85.1M  +  token & position embeddings  23.8M  =   108.9M   (12 L d^2 = 84.9M)
GPT-2 small: layers    85.1M  +  token & position embeddings  39.4M  =   124.4M   (12 L d^2 = 84.9M)
BERT-large: layers   302.3M  +  token & position embeddings  31.8M  =   334.1M   (12 L d^2 = 302.0M)
close to the published sizes (about 110M, 124M and 335-340M); the small remainder is the pooler, segment embeddings and layer norms we left out

BERT-base comes to 108.9 million and GPT-2 small to 124.4 million, close to their published 110 and 124 million; BERT-large is 334 million (published about 340). Note that the same twelve layers of width 768 hold 85 million parameters in both; the difference between them is the vocabulary table (30,522 against 50,257 tokens) and the mask. The huge language models of today are this very recipe with larger $d$, more layers and far more data, not a different kind of network.

3.3Chapter summary

• Encoder-only (BERT): bidirectional, MLM, understanding. Decoder-only (GPT, LLaMA): causal, next token, generation. Encoder-decoder: translation, summarisation.
• The mask decides the family: all-ones (BERT) against lower-triangular (GPT).
• Size $\approx\text{layers}\times12d^2+$ embeddings: measured 108.9M (BERT-base), 124.4M (GPT-2 small), 334M (BERT-large).

BERT (Devlin et al., 2018) and GPT (Radford et al., 2018) are the two pre-training recipes that made "pre-train on a lot of text, then adapt" the default in language processing. Both are stacks of the layers of Chapter 2; they differ in the mask and in the task they pre-train on.

4.1BERT: read both sides, fill the blanks

BERT is a stack of bidirectional encoder layers. It is pre-trained on two tasks. Masked language modelling (MLM): choose 15% of the input tokens at random and predict them from their context. The notes' example, "The [MASK] sat on the mat", is answerable only by using both the words before and the words after the blank ("sat on the mat" points to "cat"). A detail that matters: of the chosen 15%, only 80% are replaced by [MASK], 10% by a random word and 10% are left unchanged. Reason: [MASK] never occurs at fine-tuning time, so if the model were only ever asked to predict where it saw [MASK] it would not learn good representations of ordinary words. Check the proportions:

import random
random.seed(0)
n_tokens, vocab = 200_000, 30000
selected = mask_tok = random_tok = unchanged = 0
for _ in range(n_tokens):
    if random.random() < 0.15:                          # 15% of the tokens are chosen as prediction targets
        selected += 1
        r = random.random()
        if r < 0.8:   mask_tok += 1                     # 80%: replaced by [MASK]
        elif r < 0.9: random_tok += 1                   # 10%: replaced by a random word
        else:         unchanged += 1                    # 10%: left as it is (but still predicted)
print(f"tokens chosen as targets: {selected / n_tokens:.3f} of all tokens")
print(f"  of those, replaced by [MASK]: {mask_tok / selected:.3f}   random word: {random_tok / selected:.3f}   unchanged: {unchanged / selected:.3f}")
print(f"so [MASK] is visible in only {mask_tok / n_tokens:.3f} of all input tokens, and never appears at fine-tuning time:")
print("the 10% random / 10% unchanged cases stop the model from learning 'only predict where I see [MASK]'")

n = 512
print(f"\ntraining signal per sequence of {n} tokens: BERT predicts about {round(0.15 * n)} tokens (the masked ones), GPT predicts {n - 1} (every position): {(n - 1) / round(0.15 * n):.1f}x more")

tokens chosen as targets: 0.151 of all tokens
  of those, replaced by [MASK]: 0.798   random word: 0.099   unchanged: 0.103
so [MASK] is visible in only 0.121 of all input tokens, and never appears at fine-tuning time:
the 10% random / 10% unchanged cases stop the model from learning 'only predict where I see [MASK]'

training signal per sequence of 512 tokens: BERT predicts about 77 tokens (the masked ones), GPT predicts 511 (every position): 6.6x more

The proportions come out as 0.798 / 0.099 / 0.103, and [MASK] is visible in only 12% of the input. A cost of MLM shows in the last line: the model learns from only about 77 predicted tokens in a 512-token sequence, where a left-to-right model learns from 511, 6.6 times more training signal per sequence.

The second task is next-sentence prediction (NSP): given two sentences, decide whether the second really followed the first (50% of pairs) or was a random sentence (50%), which was meant to teach relations between sentences. (Later work, such as RoBERTa, found NSP to add little and dropped it; the notes list it as part of BERT, so know both facts.) For a downstream task you add a small task-specific head (for example a linear classifier on the [CLS] token) and fine-tune all the parameters, not just the head, unless compute forbids; this is the transfer learning of Week 4, applied to language.

4.2GPT: predict the next token

GPT is a stack of causal (decoder-only) layers, trained with the plain language-modelling objective of Week 7: predict token $t+1$ from tokens $1..t$ at every position, minimising the cross-entropy. Nothing is masked in the input; the causal mask in the attention is the only thing keeping the future hidden. To use it, you feed a prompt, sample a next token, append it and repeat (the generation loop of Week 8). Build the smallest honest version, a 2-layer, 105,000-parameter decoder-only Transformer, and train it on the toy language of Week 7:

import math, random, torch, torch.nn as nn
torch.manual_seed(0); random.seed(0); torch.set_num_threads(1)
animals, verbs, things = ["cat", "dog", "bird"], ["saw", "chased", "ate"], ["fish", "ball", "bone"]
text = "".join(f"the {random.choice(animals)} {random.choice(verbs)} the {random.choice(things)}. " for _ in range(1500))
chars = sorted(set(text)); idx = {c: i for i, c in enumerate(chars)}; V = len(chars)
data = torch.tensor([idx[c] for c in text])
floor = math.exp(3 * math.log(3) / (len(text) / 1500))
print(f"vocabulary {V} characters; the best possible perplexity for this toy language is {floor:.3f}")

class TinyGPT(nn.Module):                                   # decoder-only: token + position embeddings, causal Transformer layers, output head
    def __init__(self, d=64, heads=4, layers=2, ctx=40):
        super().__init__()
        self.tok = nn.Embedding(V, d); self.pos = nn.Embedding(ctx, d)
        block = nn.TransformerEncoderLayer(d, heads, 4 * d, dropout=0.0, batch_first=True, norm_first=True)
        self.blocks = nn.TransformerEncoder(block, layers); self.norm = nn.LayerNorm(d); self.head = nn.Linear(d, V)
    def forward(self, x):
        n = x.size(1); mask = nn.Transformer.generate_square_subsequent_mask(n)
        h = self.tok(x) + self.pos(torch.arange(n))
        return self.head(self.norm(self.blocks(h, mask=mask, is_causal=True)))

model = TinyGPT(); opt = torch.optim.AdamW(model.parameters(), lr=3e-3)
print("parameters:", sum(p.numel() for p in model.parameters()))
L = 40
def batch(B=64):
    s = torch.randint(0, len(data) - L - 1, (B,))
    return torch.stack([data[i:i + L] for i in s]), torch.stack([data[i + 1:i + L + 1] for i in s])
def ppl():
    model.eval()
    with torch.no_grad():
        x, y = batch(256); r = math.exp(nn.functional.cross_entropy(model(x).reshape(-1, V), y.reshape(-1)).item())
    model.train(); return r
print(f"before training: perplexity {ppl():.1f}")
for step in range(1, 601):
    x, y = batch()
    loss = nn.functional.cross_entropy(model(x).reshape(-1, V), y.reshape(-1))
    opt.zero_grad(); loss.backward(); nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step()
    if step in (50, 200, 600): print(f"step {step:3d}: perplexity {ppl():.3f}")

model.eval(); ids = [idx[c] for c in ". the "]
with torch.no_grad():
    for _ in range(110):
        logits = model(torch.tensor([ids[-40:]]))[0, -1]
        ids.append(torch.multinomial(torch.softmax(logits, -1), 1).item())
print("sample:", "".join(chars[i] for i in ids))

vocabulary 18 characters; the best possible perplexity for this toy language is 1.152
parameters: 104978
before training: perplexity 21.2
step  50: perplexity 1.789
step 200: perplexity 1.202
step 600: perplexity 1.192
sample: . the cat chased the ball. the dog chased the ball. the dog saw the bone. the dog chased the bone. the dog saw the b

Perplexity falls from 21.2 (worse than a uniform guess over 18 characters) to 1.79 after 50 steps, 1.20 after 200 and 1.192 after 600, within 4% of the theoretical floor of 1.152, on a par with the LSTM of Week 7 (1.19). The sample is grammatical and wholly generated by the model. Do not read this as "Transformers beat LSTMs": on a language this simple they tie. The reasons the field moved to Transformers are different, and are the ones in the earlier chapters: parallel training over positions (no step-by-step recurrence), a direct path between any two tokens for gradients and information, and good behaviour as size and data grow.

4.3BERT versus GPT

4.4Chapter summary

• BERT: bidirectional encoder; MLM (15%: 80% [MASK], 10% random, 10% unchanged; measured 0.798/0.099/0.103) + NSP; fine-tune all parameters with a task head.
• GPT: causal decoder; next-token prediction at every position (511 targets against BERT's 77 in 512 tokens); generate by feeding outputs back.
• Measured: a 105k-parameter tiny GPT reached perplexity 1.192 (floor 1.152) in 600 steps.

Can the Transformer, built for words, read pictures? The obstacle is cost. If every pixel were a token, a $224\times224$ image would be 50,176 tokens and self-attention (Week 9) would need a score matrix of $50{,}176^2=2.5$ billion entries. The Vision Transformer (ViT) (Dosovitskiy et al., 2020) avoids it with one simple move: make the tokens patches.

5.1The recipe

• Cut the image into fixed-size patches, for example $16\times16$ pixels.
• Flatten each patch into a vector ($16\times16\times3=768$ numbers).
• Project each vector with one linear layer to the model width $d$ (the patch embedding).
• Prepend a learnable [CLS] token (as in BERT) whose final output is used for classification, and add position embeddings (the patches would otherwise be an unordered set).
• Feed the sequence to a standard Transformer encoder: nothing in it is specific to images.

import torch, torch.nn as nn
torch.manual_seed(0)
img = torch.randn(1, 3, 224, 224)
patch, d = 16, 768
n_patches = (224 // patch) ** 2
print(f"a {224}x{224} image has {224 * 224:,} pixels; with {patch}x{patch} patches it becomes {n_patches} tokens ({224 // patch} x {224 // patch})")
print("each patch holds 16 x 16 x 3 =", patch * patch * 3, "numbers")

embed = nn.Conv2d(3, d, kernel_size=patch, stride=patch)             # the shortcut: one convolution with kernel = stride = patch size
tokens = embed(img).flatten(2).transpose(1, 2)                      # (1, 196, 768)
cls = nn.Parameter(torch.zeros(1, 1, d))
seq = torch.cat([cls.expand(1, -1, -1), tokens], 1)
print("patch embedding output:", tuple(tokens.shape), " + [CLS] token ->", tuple(seq.shape), "(then + position embeddings, then a standard Transformer encoder)")

# the convolution is exactly 'cut into patches, flatten, apply one linear layer'
patches = img.unfold(2, patch, patch).unfold(3, patch, patch).permute(0, 2, 3, 1, 4, 5).reshape(1, n_patches, -1)
linear = nn.Linear(patch * patch * 3, d)
with torch.no_grad():
    linear.weight.copy_(embed.weight.reshape(d, -1)); linear.bias.copy_(embed.bias)
print("conv patch embedding == flatten each patch + linear layer:", torch.allclose(linear(patches), tokens, atol=1e-4))
print("patch-embedding parameters:", sum(p.numel() for p in embed.parameters()), "= 768 * (16*16*3) + 768")

for name, n in (("one token per pixel", 224 * 224), ("one token per 16x16 patch + CLS", n_patches + 1)):
    print(f"self-attention score matrix for {name}: {n}^2 = {n * n:,} entries")

a 224x224 image has 50,176 pixels; with 16x16 patches it becomes 196 tokens (14 x 14)
each patch holds 16 x 16 x 3 = 768 numbers
patch embedding output: (1, 196, 768)  + [CLS] token -> (1, 197, 768) (then + position embeddings, then a standard Transformer encoder)
conv patch embedding == flatten each patch + linear layer: True
patch-embedding parameters: 590592 = 768 * (16*16*3) + 768
self-attention score matrix for one token per pixel: 50176^2 = 2,517,630,976 entries
self-attention score matrix for one token per 16x16 patch + CLS: 197^2 = 38,809 entries

A $224\times224$ image becomes a $14\times14$ grid, 196 patches, and with the [CLS] token a sequence of 197 tokens of width 768. The score matrix is $197^2=38{,}809$ entries instead of 2.5 billion, a reduction of 65,000 times. The shortcut in the notes is exact: Conv2d(3, 768, kernel_size=16, stride=16) with kernel and stride both equal to the patch size cuts the image into patches and applies the linear projection in a single operation, and the code confirms it equals "flatten each patch, apply one linear layer" (True), with $768\times768+768=590{,}592$ parameters.

5.2ViT versus a CNN

A CNN (Week 3) has strong built-in assumptions, inductive biases: nearby pixels matter together (locality) and the same pattern means the same thing anywhere (translation equivariance, via weight sharing). These are very good guesses about images, so a CNN learns well from small datasets. A ViT has almost none of them: its attention is global from the very first layer and it must learn from data that nearby patches are related and that a pattern is the same wherever it occurs. The consequences, as reported in the original paper and stated in the notes: on small data a ViT does worse than a comparable CNN; trained on very large data (ImageNet-21k and beyond) it matches or beats CNNs. ViT needs more data, not less. (This is a result from the literature that this course does not reproduce; the toy experiments here are too small to test it.)

5.3Chapter summary

• ViT: cut into 16x16 patches (196 for 224x224), flatten (768 numbers), linear projection, prepend [CLS], add position embeddings, standard Transformer encoder.
• Patches cut the score matrix from 2.5 billion to 38,809 entries; Conv2d(kernel = stride = patch) is the exact shortcut (verified).
• ViT has little inductive bias: worse than a CNN on small data, better on very large data.

What is likely to be asked from Week 10, with 30-second answers, deeper explanations, a lecture-or-extra tag on each question, and code you can run.

Week 10 questions are about the pieces of a Transformer, how they fit together, and the three families built from them. Each answer starts with a 30-second version. Questions are tagged From the notes or Extra.

6.1Interview heat-map: what is most likely

6.2Position, normalisation and layers

PE(p) . PE(p+k) for several starting positions p (columns) and offsets k (rows):
  k \ p     0       10      50      200
    0     8.000   8.000   8.000   8.000
    1     7.485   7.485   7.485   7.485
    2     6.368   6.368   6.368   6.368
    5     6.137   6.137   6.137   6.137
   10     3.646   3.646   3.646   3.646
   30     2.699   2.699   2.699   2.699
  100     3.487   3.487   3.487   3.487

LayerNorm equals (x - mean) / std taken over the FEATURES of each token: True
every single token now has mean -0.0 and std 1.0

change the other sequences in the batch:
  LayerNorm output of sequence 0 unchanged: True
  BatchNorm output of sequence 0 unchanged: False  (its statistics came from the whole batch)

a batch of ONE token: LayerNorm works -> torch.Size([1, 1, 8]) ; BatchNorm in train mode: ValueError: Expected more than 1 value per channel when trai
parameters: LayerNorm has 16 (gamma and beta per feature) and needs no running statistics

hand-written encoder layer matches nn.TransformerEncoderLayer: True
shapes: input (2, 5, 16)  after attention (2, 5, 16)  FFN hidden (2, 5, 64)  output (2, 5, 16) (same as the input: layers can be stacked)
position-wise FFN: ffn(x[perm]) == ffn(x)[perm]: True

sub-layers: 1) masked self-attention over the 4 target tokens (4x4), 2) cross-attention to the 6 encoder positions (4x6), 3) FFN
output shape (1, 4, 16) = one vector per target token
changing the last target token leaves outputs 0, 1, 2 unchanged: True  (no leak from the future)
changing an ENCODER token changes every decoder output (cross-attention): True

hand-written encoder layer matches nn.TransformerEncoderLayer: True
shapes: input (2, 5, 16)  after attention (2, 5, 16)  FFN hidden (2, 5, 64)  output (2, 5, 16) (same as the input: layers can be stacked)
position-wise FFN: ffn(x[perm]) == ffn(x)[perm]: True

encoder layer, d = 512:
  self-attention    1,050,624   = 4 d^2 + 4 d      = 1,050,624
  feed-forward      2,099,712   = 8 d^2 + 5 d      = 2,099,712
  layer norms           2,048   = 4 d             = 2,048
  TOTAL             3,152,384   = 12.025 d^2  (the notes say ~8 d^2; the attention and FFN parts they list add to 4 + 8 = 12 d^2)
decoder layer, d = 512: 4,204,032 = 16.037 d^2  (an extra cross-attention block: 4 + 4 + 8 = 16 d^2)
original base model, 6 encoder + 6 decoder layers (no embeddings): 44,140,544

BERT-base : layers    85.1M  +  token & position embeddings  23.8M  =   108.9M   (12 L d^2 = 84.9M)
GPT-2 small: layers    85.1M  +  token & position embeddings  39.4M  =   124.4M   (12 L d^2 = 84.9M)
BERT-large: layers   302.3M  +  token & position embeddings  31.8M  =   334.1M   (12 L d^2 = 302.0M)

6.3BERT, GPT and the families

tokens chosen as targets: 0.151 of all tokens
  of those, replaced by [MASK]: 0.798   random word: 0.099   unchanged: 0.103
so [MASK] is visible in only 0.121 of all input tokens, and never appears at fine-tuning time:
the 10% random / 10% unchanged cases stop the model from learning 'only predict where I see [MASK]'

training signal per sequence of 512 tokens: BERT predicts about 77 tokens (the masked ones), GPT predicts 511 (every position): 6.6x more

BERT (encoder-only): who can each token attend to?  1 = allowed
[[1 1 1 1 1 1]
 [1 1 1 1 1 1]
 [1 1 1 1 1 1]
 [1 1 1 1 1 1]
 [1 1 1 1 1 1]
 [1 1 1 1 1 1]]
GPT (decoder-only, causal):
[[1 0 0 0 0 0]
 [1 1 0 0 0 0]
 [1 1 1 0 0 0]
 [1 1 1 1 0 0]
 [1 1 1 1 1 0]
 [1 1 1 1 1 1]]
tokens visible to each position, BERT: [6, 6, 6, 6, 6, 6]   GPT: [1, 2, 3, 4, 5, 6]
BERT sees the right context too, which suits 'understand the sentence' tasks; GPT cannot, which is what allows it to generate left to right

BERT (encoder-only): who can each token attend to?  1 = allowed
[[1 1 1 1 1 1]

before training: perplexity 21.2
step  50: perplexity 1.789
step 200: perplexity 1.202
step 600: perplexity 1.192
sample: . the cat chased the ball. the dog chased the ball. the dog saw the bone. the dog chased the bone. the dog saw the b

6.4Vision

a 224x224 image has 50,176 pixels; with 16x16 patches it becomes 196 tokens (14 x 14)
each patch holds 16 x 16 x 3 = 768 numbers
patch embedding output: (1, 196, 768)  + [CLS] token -> (1, 197, 768) (then + position embeddings, then a standard Transformer encoder)
conv patch embedding == flatten each patch + linear layer: True
patch-embedding parameters: 590592 = 768 * (16*16*3) + 768

Week 10 on one page: chapter glance, the equations, the layer templates, the three families side by side, a checklist and corrections to the source notes.

7.1Week 10 at a glance

7.2The layer templates

7.3Three families side by side

7.4Key terms

7.5The pairs you must tell apart

7.6Last-minute checklist

7.7Corrections and additions to the source notes