Deep Learning,
Week 6

Until now every network answered a question about an input: is this a cat? what is the price? This week the network produces the input: a new face, a new sentence, a new melody. That is generative modelling, and it starts with a precise statement of the difference.

1.1Discriminative versus generative

A generative classifier decides with Bayes' theorem, $P(\text{cat}\mid x)=\dfrac{P(x\mid\text{cat})P(\text{cat})}{P(x)}$, comparing the posteriors of the classes (the $P(x)$ cancels in the comparison). Here it is on real numbers: body length of cats and dogs, learned from 1,000 animals.

import numpy as np
rng = np.random.default_rng(0)
# two classes of animals described by ONE feature, the body length in cm
cat = rng.normal(46, 5, 400); dog = rng.normal(62, 8, 600)
x = np.concatenate([cat, dog]); y = np.concatenate([np.zeros(400), np.ones(600)])

# GENERATIVE: learn P(x | class) and P(class), classify with Bayes' theorem
mu = [cat.mean(), dog.mean()]; sd = [cat.std(), dog.std()]; prior = [0.4, 0.6]
pdf = lambda v, m, s: np.exp(-0.5 * ((v - m) / s) ** 2) / (s * np.sqrt(2 * np.pi))
post_cat = lambda v: pdf(v, mu[0], sd[0]) * prior[0] / (pdf(v, mu[0], sd[0]) * prior[0] + pdf(v, mu[1], sd[1]) * prior[1])
print(f"learned: cat ~ N({mu[0]:.1f}, {sd[0]:.1f}),  dog ~ N({mu[1]:.1f}, {sd[1]:.1f}),  P(cat) = {prior[0]}")
for v in (45, 52, 55, 60):
    print(f"  P(cat | length={v}) = {post_cat(v):.3f}")

# DISCRIMINATIVE: learn P(class | x) directly (logistic regression by gradient descent)
w, b = 0.0, 0.0; xs = (x - 55) / 10
for _ in range(4000):
    p = 1 / (1 + np.exp(-(w * xs + b))); w -= 0.3 * np.mean((p - y) * xs); b -= 0.3 * np.mean(p - y)
boundary_disc = 55 + 10 * (-b / w)
grid = np.linspace(40, 75, 3501); boundary_gen = grid[np.argmin(abs(post_cat(grid) - 0.5))]
print(f"decision boundary: generative {boundary_gen:.1f} cm, discriminative {boundary_disc:.1f} cm")

# ONLY the generative model can create new data, and score how plausible a point is
print("new synthetic cats:", np.round(rng.normal(mu[0], sd[0], 5), 1))
print("log-density of a 46 cm cat:", round(float(np.log(pdf(46, mu[0], sd[0]))), 2), "  of a 100 cm 'cat':", round(float(np.log(pdf(100, mu[0], sd[0]))), 2), "(anomaly detection)")

learned: cat ~ N(45.8, 5.0),  dog ~ N(61.6, 7.7),  P(cat) = 0.4
  P(cat | length=45) = 0.911
  P(cat | length=52) = 0.507
  P(cat | length=55) = 0.213
  P(cat | length=60) = 0.018
decision boundary: generative 52.1 cm, discriminative 51.6 cm
new synthetic cats: [51.7 47.3 46.8 47.1 39. ]
log-density of a 46 cm cat: -2.52   of a 100 cm 'cat': -61.82 (anomaly detection)

Three things to read off. (1) The generative route classifies correctly: at 45 cm the cat posterior is 0.91, at 60 cm only 0.02. (2) Its decision boundary (52.1 cm) is almost identical to the one logistic regression learns directly (51.6 cm), because here the two curves are close to Gaussians; the discriminative model got the same answer without modelling any curve. (3) Only the generative model can sample new cats and can score a point: a 100 cm "cat" has a log-density of $-61.8$ against $-2.5$ for a typical one, which is how generative models do anomaly detection. Generative models can also fill in missing data (in-painting).

1.2Why modelling images directly is hopeless

import math
n_pixels = 100 * 100
digits = int(n_pixels * math.log10(256)) + 1       # every pixel takes one of 256 values: 256^10000 images
print("a 100x100 greyscale image has", n_pixels, "pixels")
print("number of possible images: 256^10000 has", digits, "digits")
print("for scale: the number of atoms in the observable universe has about 80 digits")
import numpy as np
rng = np.random.default_rng(0)
noise = rng.integers(0, 256, (100, 100))
print("a uniformly random image has pixel std", round(float(noise.std()), 1), "(pure static); almost every point in pixel space looks like this")

a 100x100 greyscale image has 10000 pixels
number of possible images: 256^10000 has 24083 digits
for scale: the number of atoms in the observable universe has about 80 digits
a uniformly random image has pixel std 74.4 (pure static); almost every point in pixel space looks like this

Writing down $P(x_1,\dots,x_{10000})$ for a small 100x100 image means assigning a probability to a number with 24,083 digits of possibilities, nearly all of which are static. Real images occupy a vanishingly small corner of that space. So generative models do not model pixels one by one; they find a much smaller set of hidden numbers that explains the image, the latent space of the next chapter.

1.3A taxonomy of generative models

1.4Chapter summary

• Discriminative: $P(y\mid x)$, boundary only. Generative: $P(x\mid y),P(y)$, can sample, score and classify via Bayes.
• Measured: Bayes boundary 52.1 cm against logistic regression 51.6 cm; a 100 cm "cat" scored $-61.8$ against $-2.5$.
• Pixels are hopeless to model directly (24,083-digit space); GANs are implicit density models.

The way out of the pixel jungle is the idea behind every modern generative model: real data has far fewer degrees of freedom than it has numbers.

2.1The manifold idea

Take 8x8 pictures (64 numbers each) of a single bright blob. Every such picture is fixed by two numbers: where the blob sits, $(c_x,c_y)$. The 64 pixel values are a redundant description. The set of all such pictures is a thin two-dimensional surface, a manifold, curled up inside 64-dimensional pixel space. The two numbers are the latent variables; the space they span is the latent space. Real photographs are the same story with, say, a few hundred latent factors (pose, lighting, identity, expression) inside millions of pixels. JPEG works because of the same redundancy.

2.2Before deep learning: PCA and factor analysis

PCA finds the best linear projection to fewer dimensions: the directions of largest variance. Factor analysis assumes the data are a linear mixture of a few hidden factors plus noise, $x=Wz+\mu+\varepsilon$. Both are linear latent-variable models. Test PCA on the blob pictures:

import numpy as np
# 8x8 images of a bright blob. Only TWO numbers (its x and y position) decide every image,
# but each image has 64 pixels.
def blob(cx, cy, size=8, width=1.1):
    yy, xx = np.mgrid[0:size, 0:size]
    return np.exp(-((xx - cx) ** 2 + (yy - cy) ** 2) / (2 * width ** 2)).ravel()

rng = np.random.default_rng(0)
latent = rng.uniform(1.5, 5.5, (500, 2))
X = np.stack([blob(a, b) for a, b in latent])
Xc = X - X.mean(0)
s = np.linalg.svd(Xc, compute_uv=False) ** 2
ratio = s / s.sum()
print("images: 500 x 64 pixels, generated from only 2 latent numbers (cx, cy)")
print("share of variance by PCA component:", np.round(ratio[:8], 3).tolist())
print("cumulative, first 2 / 4 / 8 components:", [round(float(ratio[:k].sum()), 3) for k in (2, 4, 8)])

# Linear (PCA) compression needs more than 2 numbers although the data has 2 degrees of freedom: the set of
# blob images is a CURVED 2-D surface inside 64-D, not a flat plane. That is why generative models learn
# a NON-linear map from a small latent space to the images.
a, b = blob(1.5, 1.5), blob(5.5, 5.5)
mid_pixel = (a + b) / 2                                   # average the two images pixel by pixel
mid_latent = blob(3.5, 3.5)                               # move halfway in the latent (position) space
print("halfway in PIXEL space : max brightness", round(float(mid_pixel.max()), 2), "(two half-bright ghosts, not a blob in the middle)")
print("halfway in LATENT space: max brightness", round(float(mid_latent.max()), 2), "(one full blob, in the middle)")

images: 500 x 64 pixels, generated from only 2 latent numbers (cx, cy)
share of variance by PCA component: [0.343, 0.291, 0.116, 0.101, 0.066, 0.025, 0.023, 0.01]
cumulative, first 2 / 4 / 8 components: [0.635, 0.852, 0.975]
halfway in PIXEL space : max brightness 0.41 (two half-bright ghosts, not a blob in the middle)
halfway in LATENT space: max brightness 0.81 (one full blob, in the middle)

The pictures have exactly 2 degrees of freedom, yet the first two principal components capture only 63.5% of the variance; it takes 4 components for 85% and 8 for 97.5%. A flat plane cannot lie along a curved surface, so a linear method keeps needing more directions. This is precisely the limit that the generative models of this course remove: they learn a non-linear map from a small latent space to the image, which is what a neural network is for.

2.3Why the latent space is the whole point

The last two lines of the output are the picture above. Averaging two blob images pixel by pixel gives two half-bright ghosts (maximum brightness 0.41), which is not a valid image of the family. Moving halfway in the latent space (the blob's position) gives one full-bright blob in the middle (0.81). A generative model gives you this: random noise or a point in latent space goes in, a valid image comes out, and walking through latent space morphs one valid image smoothly into another. That is what makes sampling, interpolation and editing possible.

2.4Chapter summary

• Images live on a low-dimensional manifold; a few latent variables describe them (the blob: 2 numbers for 64 pixels).
• PCA/factor analysis are linear: 63.5% variance in 2 components although the true dimension is 2.
• Latent interpolation gives valid in-between images (0.81); pixel averaging gives ghosts (0.41).

A Generative Adversarial Network (Goodfellow et al., 2014) trains a generator without ever writing down a density: it sets up a game between two networks and lets competition do the teaching.

3.1The two players

• Generator $G(z;\theta_g)$: takes random noise $z\sim p_z$ and outputs a fake sample. Its goal: be mistaken for real.
• Discriminator $D(x;\theta_d)$: takes a sample and outputs the probability $D(x)\in(0,1)$ that it is real. Its goal: real $\to1$, fake $\to0$.
• The generator never sees a real image. Its only teacher is the gradient that flows back through $D$.

3.2The minimax objective

Read the two terms as a score for D. The first, $\log D(x)$ on real data, is largest (0) when $D(x)=1$. The second, $\log(1-D(G(z)))$ on fakes, is largest (0) when $D(G(z))=0$. So D maximises $V$: it is the score of a good critic. G only appears in the second term, and G minimises $V$: it wants $D(G(z))$ to be large, to drive that term to $-\infty$. Both directions matter: D maximises, G minimises, never both minimising.

In code, $V$ is just binary cross-entropy. With real label 1 and fake label 0, the discriminator's loss is $L_D=-\big[\mathbb{E}\log D(x)+\mathbb{E}\log(1-D(G(z)))\big]$ (minimised, which is the same as maximising $V$). Evaluate the losses at the very start, when D knows nothing and says 0.5 everywhere:

import torch
bce = torch.nn.functional.binary_cross_entropy_with_logits
B = 1000
d_real = torch.zeros(B); d_fake = torch.zeros(B)                    # logit 0 means D outputs 0.5 for everything
loss_D = bce(d_real, torch.ones(B)) + bce(d_fake, torch.zeros(B))   # -[log D(x) + log(1 - D(G(z)))]
loss_G = bce(d_fake, torch.ones(B))                                 # -log D(G(z))   (non-saturating)
print("at initialisation D(x) = 0.5 everywhere:")
print("  D loss =", round(loss_D.item(), 4), " = 2 log 2 =", round(2 * torch.log(torch.tensor(2.0)).item(), 4))
print("  G loss =", round(loss_G.item(), 4), " = log 2   =", round(torch.log(torch.tensor(2.0)).item(), 4))
for name, dr, df in (("D wins easily ", 6.0, -6.0), ("D is fooled   ", -3.0, 3.0)):
    r = torch.full((B,), dr); f = torch.full((B,), df)
    print(f"{name}: D loss = {(bce(r, torch.ones(B)) + bce(f, torch.zeros(B))).item():.4f}, G loss = {bce(f, torch.ones(B)).item():.4f}")

at initialisation D(x) = 0.5 everywhere:
  D loss = 1.3863  = 2 log 2 = 1.3863
  G loss = 0.6931  = log 2   = 0.6931
D wins easily : D loss = 0.0050, G loss = 6.0025
D is fooled   : D loss = 6.0972, G loss = 0.0486

At initialisation $L_D=-[\log0.5+\log0.5]=2\log2=\mathbf{1.386}$, and the generator's loss is $\log2=0.693$. Those numbers matter, because they tell you how a healthy run looks: a discriminator loss that falls toward 0 means D has won and G is failing (second row: D loss 0.005, G loss 6.0); a loss that stays near 1.386 means the two are balanced. The notes say "D loss decreases as D improves" and stop there; the full picture is that at the ideal equilibrium the D loss climbs back to 1.386.

3.3The best possible discriminator

Fix $G$ (so the fake distribution $p_g$ is fixed) and ask what the best $D$ is. At each point $x$ the objective contributes $p_{\text{data}}(x)\log D+p_g(x)\log(1-D)$. Setting the derivative in $D$ to zero gives

That is a probability: "of all the ways this $x$ could have arisen, what fraction is real?". Put $D^*$ back into $V$ and something beautiful happens: the objective becomes a distance between the two distributions,

import numpy as np
# two discrete distributions over 5 points
p_data = np.array([0.10, 0.40, 0.30, 0.15, 0.05])
p_g    = np.array([0.30, 0.10, 0.20, 0.10, 0.30])
D_star = p_data / (p_data + p_g)                                   # the best possible discriminator
V = lambda D, pg: float(np.sum(p_data * np.log(D)) + np.sum(pg * np.log(1 - D)))
m = (p_data + p_g) / 2
kl = lambda a, b: float(np.sum(a * np.log(a / b)))
JS = 0.5 * kl(p_data, m) + 0.5 * kl(p_g, m)
print("D*(x) =", np.round(D_star, 3).tolist(), " (high where real data is more likely than fake)")
print(f"V(D*, G)              = {V(D_star, p_g):.4f}")
print(f"-log 4 + 2 * JS(p_data || p_g) = {-np.log(4) + 2 * JS:.4f}   (JS = {JS:.4f})  -> they match")
print("a generator that matches the data exactly:")
D_eq = p_data / (p_data + p_data)
print("  D* =", D_eq.tolist(), "  V =", round(V(D_eq, p_data), 4), "  = -log 4 =", round(float(-np.log(4)), 4))

D*(x) = [0.25, 0.8, 0.6, 0.6, 0.143]  (high where real data is more likely than fake)
V(D*, G)              = -1.1234
-log 4 + 2 * JS(p_data || p_g) = -1.1234   (JS = 0.1314)  -> they match
a generator that matches the data exactly:
  D* = [0.5, 0.5, 0.5, 0.5, 0.5]   V = -1.3863   = -log 4 = -1.3863

The numbers agree to four decimals ($-1.1234$ both ways). So, with a perfect discriminator, the generator is minimising the Jensen-Shannon divergence between the data and its own distribution. JS is never negative and is 0 only when $p_g=p_{\text{data}}$, which is exactly where $V=-\log4=-1.386$ and $D^*=\tfrac12$ everywhere (the last lines): the critic can no longer tell real from fake, so the best it can do is guess. This is the Nash equilibrium, the target of the whole game.

3.4Chapter summary

• $\min_G\max_D\ \mathbb{E}\log D(x)+\mathbb{E}\log(1-D(G(z)))$: D maximises, G minimises; G never sees real data.
• At init D outputs 0.5: $L_D=2\log2=1.386$, $L_G=\log2=0.693$. D loss near 0 means D is winning.
• $D^*=p_{\text{data}}/(p_{\text{data}}+p_g)$; then $V=-\log4+2\,\mathrm{JS}$; equilibrium $p_g=p_{\text{data}}$, $D^*=\tfrac12$.

Knowing the objective is half the job. Training a GAN is an unusual optimisation: two networks, two losses, and each one's loss changes whenever the other moves. This chapter covers the training loop, the one trick that makes it work in practice, and a complete GAN you can read line by line and watch learn.

4.1The alternating loop

• Phase 1, train D (G frozen): show D a batch of real data (target 1) and a batch of fakes from G (target 0); one gradient step on $L_D$.
• Phase 2, train G (D frozen): make fresh fakes, ask D about them, and update only G so that D says "real" (target 1 for the fakes).
• Repeat. Some GANs take $k>1$ D steps per G step, to keep the critic ahead.

4.2The saturation problem and the non-saturating loss

Early in training the generator is awful, so D is sure its samples are fake: $D(G(z))\approx0$. The original generator objective is to minimise $\log(1-D(G(z)))$. Look at its gradient there. Let $a$ be D's logit (so $D=\sigma(a)$); this table compares the gradient size of the original loss with the alternative $-\log D(G(z))$:

import torch
print("D is confident that a sample is fake when its logit a is very negative. How strong is G's learning signal?")
print("  logit a   D(G(z))   gradient of  log(1-D)  [original]   gradient of  -log D  [non-saturating]")
for a in (-6.0, -4.0, -2.0, 0.0, 2.0):
    t = torch.tensor(a, requires_grad=True)
    torch.log(1 - torch.sigmoid(t)).backward(); g_orig = t.grad.item()          # G minimises this
    t2 = torch.tensor(a, requires_grad=True)
    (-torch.log(torch.sigmoid(t2))).backward(); g_ns = t2.grad.item()          # G minimises this instead
    print(f"  {a:+6.1f}    {torch.sigmoid(torch.tensor(a)).item():.4f}    {g_orig:+20.4f}              {g_ns:+20.4f}")
print("both push D(G(z)) up (negative gradient = go up in logit), but when D wins the original loss is almost silent")

D is confident that a sample is fake when its logit a is very negative. How strong is G's learning signal?
  logit a   D(G(z))   gradient of  log(1-D)  [original]   gradient of  -log D  [non-saturating]
    -6.0    0.0025                 -0.0025                           -0.9975
    -4.0    0.0180                 -0.0180                           -0.9820
    -2.0    0.1192                 -0.1192                           -0.8808
    +0.0    0.5000                 -0.5000                           -0.5000
    +2.0    0.8808                 -0.8808                           -0.1192
both push D(G(z)) up (negative gradient = go up in logit), but when D wins the original loss is almost silent

At $a=-6$ the original loss has a gradient of 0.0025, the alternative 0.9975: 400 times more. The original loss is saturating: exactly when G is worst and most needs a signal, the signal vanishes (the sigmoid is flat out there). Both losses push in the same direction and share the same optimum, so the fix costs nothing: train G to maximise $\log D(G(z))$, i.e. minimise $L_G=-\mathbb{E}\log D(G(z))$, the non-saturating loss. In code that is simply "binary cross-entropy of D's verdict on the fakes against the label 1".

4.3A complete GAN, run for real

The smallest GAN that shows everything: the real data is a one-dimensional bell curve, $\mathcal{N}(4,1.25^2)$; G is a small MLP turning 4 noise numbers into one number; D is a small MLP turning a number into a real/fake logit. The generator begins at $\approx\mathcal{N}(0,0.06^2)$, far from the data, and must be pulled there only through D's feedback. Read the loop: it is the two-phase procedure above, line for line.

import torch, torch.nn as nn
torch.set_num_threads(1)
bce = nn.functional.binary_cross_entropy_with_logits

def train_gan(real_batch, steps, seed, zdim=4, hid=32, B=256, report=()):
    torch.manual_seed(seed)
    G = nn.Sequential(nn.Linear(zdim, hid), nn.ReLU(), nn.Linear(hid, hid), nn.ReLU(), nn.Linear(hid, 1))
    D = nn.Sequential(nn.Linear(1, hid), nn.LeakyReLU(0.2), nn.Linear(hid, hid), nn.LeakyReLU(0.2), nn.Linear(hid, 1))
    optG = torch.optim.Adam(G.parameters(), lr=1e-3, betas=(0.5, 0.999))
    optD = torch.optim.Adam(D.parameters(), lr=1e-3, betas=(0.5, 0.999))
    ones, zeros, history = torch.ones(B), torch.zeros(B), []
    for step in range(steps + 1):
        if step in report:
            with torch.no_grad():
                fake = G(torch.randn(5000, zdim)).squeeze(1)
            history.append((step, fake.mean().item(), fake.std().item(), (fake > 0).float().mean().item()))
        # phase 1: train D with G frozen (detach), real -> 1, fake -> 0
        x = real_batch(B).unsqueeze(1); fake = G(torch.randn(B, zdim)).detach()
        loss_D = bce(D(x).squeeze(1), ones) + bce(D(fake).squeeze(1), zeros)
        optD.zero_grad(); loss_D.backward(); optD.step()
        # phase 2: train G with D frozen, fool D: fake -> 1 (the non-saturating loss)
        loss_G = bce(D(G(torch.randn(B, zdim))).squeeze(1), ones)
        optG.zero_grad(); loss_G.backward(); optG.step()
    return history, loss_D.item()

gaussian = lambda n: 4 + 1.25 * torch.randn(n)
hist, loss_D = train_gan(gaussian, 3000, seed=0, report=(0, 250, 500, 1000, 2000, 3000))
print("real data ~ N(4, 1.25): the generator starts at about N(0, 0.06)")
print("  step   generated mean   generated std")
for step, m, s, _ in hist:
    print(f"  {step:4d}   {m:12.2f}   {s:12.2f}")
print(f"final D loss = {loss_D:.3f}   (the equilibrium value is 2 log 2 = 1.386)")

real data ~ N(4, 1.25): the generator starts at about N(0, 0.06)
  step   generated mean   generated std
     0           0.13           0.06
   250           4.02           1.12
   500           4.01           1.04
  1000           4.03           1.44
  2000           3.96           1.01
  3000           3.97           1.30
final D loss = 1.386   (the equilibrium value is 2 log 2 = 1.386)

In 250 steps the generated mean jumps from 0.13 to 4.02, the real mean, and the standard deviation from 0.06 to 1.12 (real: 1.25): the generator learned the distribution without ever seeing a single real number, only D's verdicts. Two honest observations. The standard deviation wobbles between 1.01 and 1.44 and never settles exactly: the game is a continual chase, not a descent into a minimum (Chapter 6 shows why). And the final D loss is 1.386, the equilibrium value $2\log2$: D is reduced to guessing, which is what success looks like.

4.4Practical settings

• Adam with a small learning rate ($2\times10^{-4}$ is the DCGAN default) and $\beta_1=0.5$ instead of 0.9: heavy momentum makes the two-player game oscillate (Week 5, GAN row).
• Use binary_cross_entropy_with_logits on raw logits, not sigmoid then BCE, for numerical stability.
• Watch the samples, not just the losses: because the losses depend on each other, a GAN's loss curves say very little about quality.

4.5Chapter summary

• Alternate: D on real vs detached fakes; then G (D frozen, only G's optimiser steps).
• Use the non-saturating loss $-\log D(G(z))$: gradient 0.9975 against 0.0025 when D is confident.
• Measured: the toy GAN reached the data (mean 4.02, std 1.12) in 250 steps; final D loss 1.386.

The toy GAN used MLPs on a single number. For images the same game is played with convolutional networks, and two designs are the ones every exam asks about: DCGAN, the recipe that made image GANs trainable, and StyleGAN, the one that made them photorealistic and controllable.

5.1DCGAN: the recipe

• Strided convolutions instead of pooling. The network learns its own down-sampling (D) and up-sampling (G, with transposed convolutions).
• Batch normalisation in both networks except the generator's output layer and the discriminator's input layer.
• No fully connected layers: convolutional all the way.
• ReLU in G (output Tanh, images scaled to $[-1,1]$); LeakyReLU in D (keeps a small gradient for negative values, so D never goes silent).

The reason for the batch-norm exceptions: normalising the very last layer of G would strip its freedom to produce the output's true mean and scale, and normalising D's first layer would erase statistics of the raw input that D needs. Build the actual 64x64 networks and watch the shapes:

import torch, torch.nn as nn
nz, nf = 100, 64
G = nn.Sequential(                                                    # noise (100) -> 3x64x64 image
    nn.ConvTranspose2d(nz, nf * 8, 4, 1, 0, bias=False), nn.BatchNorm2d(nf * 8), nn.ReLU(True),        # 4x4
    nn.ConvTranspose2d(nf * 8, nf * 4, 4, 2, 1, bias=False), nn.BatchNorm2d(nf * 4), nn.ReLU(True),    # 8x8
    nn.ConvTranspose2d(nf * 4, nf * 2, 4, 2, 1, bias=False), nn.BatchNorm2d(nf * 2), nn.ReLU(True),    # 16x16
    nn.ConvTranspose2d(nf * 2, nf, 4, 2, 1, bias=False), nn.BatchNorm2d(nf), nn.ReLU(True),            # 32x32
    nn.ConvTranspose2d(nf, 3, 4, 2, 1, bias=False), nn.Tanh())                                         # 64x64, NO BatchNorm, Tanh
D = nn.Sequential(                                                    # image -> real/fake logit
    nn.Conv2d(3, nf, 4, 2, 1, bias=False), nn.LeakyReLU(0.2, True),                                    # 32x32, NO BatchNorm on the input layer
    nn.Conv2d(nf, nf * 2, 4, 2, 1, bias=False), nn.BatchNorm2d(nf * 2), nn.LeakyReLU(0.2, True),       # 16x16
    nn.Conv2d(nf * 2, nf * 4, 4, 2, 1, bias=False), nn.BatchNorm2d(nf * 4), nn.LeakyReLU(0.2, True),   # 8x8
    nn.Conv2d(nf * 4, nf * 8, 4, 2, 1, bias=False), nn.BatchNorm2d(nf * 8), nn.LeakyReLU(0.2, True),   # 4x4
    nn.Conv2d(nf * 8, 1, 4, 1, 0, bias=False))                                                         # 1x1: one number per image

x = torch.randn(2, nz, 1, 1)
print("generator shapes:")
for layer in G:
    x = layer(x)
    if isinstance(layer, (nn.ConvTranspose2d, nn.Tanh)):
        print("  ", layer.__class__.__name__.ljust(16), tuple(x.shape))
y = x
print("discriminator shapes:")
for layer in D:
    y = layer(y)
    if isinstance(layer, nn.Conv2d):
        print("  ", layer.__class__.__name__.ljust(16), tuple(y.shape))
count = lambda m: sum(p.numel() for p in m.parameters())
print(f"parameters: generator {count(G):,}, discriminator {count(D):,}")
print("pooling layers:", sum(isinstance(m, (nn.MaxPool2d, nn.AvgPool2d)) for m in list(G) + list(D)), "  fully connected layers:", sum(isinstance(m, nn.Linear) for m in list(G) + list(D)))

generator shapes:
   ConvTranspose2d  (2, 512, 4, 4)
   ConvTranspose2d  (2, 256, 8, 8)
   ConvTranspose2d  (2, 128, 16, 16)
   ConvTranspose2d  (2, 64, 32, 32)
   ConvTranspose2d  (2, 3, 64, 64)
   Tanh             (2, 3, 64, 64)
discriminator shapes:
   Conv2d           (2, 64, 32, 32)
   Conv2d           (2, 128, 16, 16)
   Conv2d           (2, 256, 8, 8)
   Conv2d           (2, 512, 4, 4)
   Conv2d           (2, 1, 1, 1)
parameters: generator 3,576,704, discriminator 2,765,568
pooling layers: 0   fully connected layers: 0

The generator turns a $100\times1\times1$ noise tensor into $3\times64\times64$ by one 4x4 layer and four doublings (4, 8, 16, 32, 64 pixels), each time halving the channels (512, 256, 128, 64, 3); the discriminator is its mirror image, ending in a single number per image. Totals: 3.58 million parameters in G, 2.77 million in D, and, as the last line confirms, zero pooling and zero fully connected layers.

The size of a transposed convolution

import itertools, torch, torch.nn as nn
formula = lambda n, k, s, p, op=0: (n - 1) * s - 2 * p + k + op     # transposed-convolution output size
bad = 0; total = 0
for n, k, s, p in itertools.product((1, 4, 7, 16), (2, 3, 4, 5), (1, 2, 3), (0, 1, 2)):
    if 2 * p > (n - 1) * s + k - 1:                                   # padding too large for this layer
        continue
    out = nn.ConvTranspose2d(1, 1, k, s, p)(torch.zeros(1, 1, n, n)).shape[-1]
    total += 1; bad += out != formula(n, k, s, p)
print(f"checked {total} combinations against torch: {bad} mismatches")
print("the DCGAN doubling layer (kernel 4, stride 2, padding 1):", [f"{n}->{formula(n, 4, 2, 1)}" for n in (4, 8, 16, 32)])
print("the first layer (kernel 4, stride 1, padding 0) turns a 1x1 noise vector into:", f"1->{formula(1, 4, 1, 0)}")

checked 132 combinations against torch: 0 mismatches
the DCGAN doubling layer (kernel 4, stride 2, padding 1): ['4->8', '8->16', '16->32', '32->64']
the first layer (kernel 4, stride 1, padding 0) turns a 1x1 noise vector into: 1->4

This is the Week 3 convolution formula run backwards. The formula matched PyTorch on all 132 settings tried. With kernel 4, stride 2, padding 1 it gives $(n-1)\cdot2-2+4=2n$: exactly doubling, which is why DCGAN uses that setting everywhere. (A transposed convolution is the gradient-of-a-convolution operation used as a layer: it spreads each input value over a $k\times k$ patch of the output, and is learned, unlike plain interpolation.)

5.2StyleGAN: control over the image

• Mapping network. A stack of fully connected layers turns the noise $z$ into an intermediate latent $w$. The aim is disentanglement: $z$ must follow a fixed Gaussian, which forces entangled factors; $w$ is free to be shaped so that different factors (age, pose, hair) vary more independently. It is not "the same latent with a new name".
• Style injection. $w$ is passed through a small learned layer at every resolution to produce that block's style, applied with AdaIN. Coarse blocks (4x4, 8x8) control pose and face shape, fine blocks control colour and texture.
• Progressive growing (in the original): train at 4x4, then add layers for 8x8, 16x16, and so on up to 1024x1024, so the network learns coarse structure first. It helps stability and speed at high resolution.

AdaIN, in numbers

Adaptive Instance Normalisation: normalise each channel of each image on its own (mean 0, std 1, as in batch norm but within one image), then give it a scale $s_c$ and shift $b_c$ computed from the style. The content's own statistics are erased and replaced by the style's:

import torch
torch.manual_seed(0)
def adain(content, scale, shift, eps=1e-5):
    # normalise EACH channel of EACH image on its own, then give it the style's scale and shift
    mean = content.mean(dim=(2, 3), keepdim=True); std = content.std(dim=(2, 3), keepdim=True)
    return scale[:, :, None, None] * (content - mean) / (std + eps) + shift[:, :, None, None]

content = torch.randn(1, 3, 8, 8) * torch.tensor([1.0, 5.0, 0.2]).view(1, 3, 1, 1) + torch.tensor([0.0, 10.0, -2.0]).view(1, 3, 1, 1)
print("content feature maps, per channel (mean, std):", [(round(m, 2), round(s, 2)) for m, s in zip(content.mean((0, 2, 3)).tolist(), content.std((0, 2, 3)).tolist())])
scale = torch.tensor([[2.0, 0.5, 1.0]]); shift = torch.tensor([[1.0, -1.0, 0.0]])    # the "style": would come from w
out = adain(content, scale, shift)
print("after AdaIN, per channel (mean, std)           :", [(round(m, 2), round(s, 2)) for m, s in zip(out.mean((0, 2, 3)).tolist(), out.std((0, 2, 3)).tolist())])
print("requested style (shift, scale)                 :", list(zip(shift[0].tolist(), scale[0].tolist())))
print("the content's own statistics are erased and replaced by the style's; its spatial pattern is kept")

content feature maps, per channel (mean, std): [(0.09, 1.05), (10.06, 5.21), (-2.01, 0.16)]
after AdaIN, per channel (mean, std)           : [(1.0, 2.0), (-1.0, 0.5), (-0.0, 1.0)]
requested style (shift, scale)                 : [(1.0, 2.0), (-1.0, 0.5), (0.0, 1.0)]
the content's own statistics are erased and replaced by the style's; its spatial pattern is kept

The three content channels had very different means and spreads (10.06 with std 5.21, $-2.01$ with 0.16, and so on); after AdaIN each carries exactly the requested style (mean 1.0 and std 2.0; $-1.0$ and 0.5; 0 and 1.0). The pattern inside each channel (where it is bright or dark) survives, which is why the style changes "how it looks" without changing "what is there".

5.3Chapter summary

• DCGAN: strided/transposed convolutions, batch norm except G's output and D's input, no FC or pooling, ReLU+Tanh in G, LeakyReLU in D.
• Transposed conv: $n_{\text{out}}=(n-1)s-2p+k$; kernel 4, stride 2, padding 1 doubles the size. G 3.58M, D 2.77M parameters.
• StyleGAN: mapping network $z\to w$ (disentangle), AdaIN style at every block, progressive growing.

GANs produce the sharpest images of their generation and are also notoriously hard to train. The failures are not bad luck; each has a clear mechanism, and each mechanism has a standard fix.

6.1Mode collapse

Mode collapse: the generator produces only one (or a few) kinds of output, however many it could. On MNIST digits it would draw only 1s. The intuition from the notes: G finds one safe output that fools the current D and has no reason to leave it, because the loss never asks for variety, only for "looks real". Each sample can look perfect while the set of samples is nearly useless. Reproduce it in the smallest possible setting: real data with two modes, half the samples near $-3$ and half near $+3$:

import torch, torch.nn as nn
torch.set_num_threads(1)
bce = nn.functional.binary_cross_entropy_with_logits

def train_gan(real_batch, steps, seed, zdim=4, hid=32, B=256, report=()):
    torch.manual_seed(seed)
    G = nn.Sequential(nn.Linear(zdim, hid), nn.ReLU(), nn.Linear(hid, hid), nn.ReLU(), nn.Linear(hid, 1))
    D = nn.Sequential(nn.Linear(1, hid), nn.LeakyReLU(0.2), nn.Linear(hid, hid), nn.LeakyReLU(0.2), nn.Linear(hid, 1))
    optG = torch.optim.Adam(G.parameters(), lr=1e-3, betas=(0.5, 0.999))
    optD = torch.optim.Adam(D.parameters(), lr=1e-3, betas=(0.5, 0.999))
    ones, zeros, history = torch.ones(B), torch.zeros(B), []
    for step in range(steps + 1):
        if step in report:
            with torch.no_grad():
                fake = G(torch.randn(5000, zdim)).squeeze(1)
            history.append((step, fake.mean().item(), fake.std().item(), (fake > 0).float().mean().item()))
        x = real_batch(B).unsqueeze(1); fake = G(torch.randn(B, zdim)).detach()
        loss_D = bce(D(x).squeeze(1), ones) + bce(D(fake).squeeze(1), zeros)
        optD.zero_grad(); loss_D.backward(); optD.step()
        loss_G = bce(D(G(torch.randn(B, zdim))).squeeze(1), ones)
        optG.zero_grad(); loss_G.backward(); optG.step()
    return history

def two_modes(n):                                     # half the data near -3, half near +3
    side = (torch.rand(n) < 0.5).float() * 2 - 1
    return 3 * side + 0.5 * torch.randn(n)

steps = (250, 500, 1000, 1500, 2000, 3000)
print("real data: 50% of samples near -3 and 50% near +3, so the fraction of samples above 0 should be 0.50")
print("fraction of generated samples above 0 (0.00 = all in the left mode, 1.00 = all in the right mode):")
print("  seed " + "".join(f"{s:>7d}" for s in steps) + "      std at 3000")
for seed in (0, 3, 4):
    h = train_gan(two_modes, 3000, seed, report=steps)
    print(f"  {seed:4d} " + "".join(f"{r[3]:7.2f}" for r in h) + f"      {h[-1][2]:.2f}  (real std 3.04)")

real data: 50% of samples near -3 and 50% near +3, so the fraction of samples above 0 should be 0.50
fraction of generated samples above 0 (0.00 = all in the left mode, 1.00 = all in the right mode):
  seed     250    500   1000   1500   2000   3000      std at 3000
     0    0.00   0.00   0.00   0.47   0.47   0.46      3.01  (real std 3.04)
     3    1.00   1.00   1.00   1.00   1.00   0.52      3.02  (real std 3.04)
     4    1.00   1.00   1.00   1.00   1.00   0.49      3.11  (real std 3.04)

The collapse is textbook. In all three runs, at steps 250, 500 and 1000 the fraction of samples above zero is exactly 0.00 (seed 0, the left mode) or 1.00 (seeds 3 and 4, the right mode): the generator has put everything on a single mode, and a different one in different runs, which is the "one safe output". The standard deviation then jumps to $\approx3$ (the real value is 3.04) when it covers both. An honest reading: in this tiny 1-D problem the collapse is temporary. The usual explanation is that D eventually learns "that mode has too many samples", pushes G off it, and G finds the other mode (I did not isolate this in the run). In large image GANs the same cat-and-mouse can persist for the whole run, or hop forever between a few modes. The sign that you are collapsed is not the loss; it is the diversity of the samples.

6.2Non-convergence: the game does not settle

In ordinary training the loss goes down and the weights settle. A GAN is a game: each player's best move depends on the other's, and simultaneous gradient steps do not have to reach the equilibrium. The simplest game that shows it: $\min_x\max_y\,xy$, whose only equilibrium is $(0,0)$. Each player takes a gradient step at the same time:

import numpy as np
# the simplest possible game: min over x, max over y of  V = x*y.  The only equilibrium is (0, 0).
# Gradient descent for the minimiser, gradient ASCENT for the maximiser, both moving at once.
eta = 0.1
x, y = 1.0, 1.0
print("step   x        y        distance from the equilibrium")
for t in range(0, 101):
    if t in (0, 10, 25, 50, 100):
        print(f"{t:4d}  {x:+7.3f}  {y:+7.3f}   {np.hypot(x, y):7.3f}")
    x, y = x - eta * y, y + eta * x                  # simultaneous update: dV/dx = y, dV/dy = x
print("each step multiplies the distance by sqrt(1 + eta^2) =", round(float(np.sqrt(1 + eta ** 2)), 5), " (>1): it spirals OUT, never in")

step   x        y        distance from the equilibrium
   0   +1.000   +1.000     1.414
  10   -0.312   +1.453     1.486
  25   -1.587   -0.216     1.602
  50   +1.579   -0.892     1.814
 100   -0.560   -2.257     2.326
each step multiplies the distance by sqrt(1 + eta^2) = 1.00499  (>1): it spirals OUT, never in

The pair circles the equilibrium and each step multiplies the distance by $\sqrt{1+\eta^2}$ (1.00499 here): from 1.41 to 2.33 in 100 steps. It never converges. Real GANs are far more complicated than $xy$, but the same rotational dynamics are why their losses oscillate and why the notes say they "never reach the Nash equilibrium in practice". Remedies include momentum choices ($\beta_1=0.5$), a smaller learning rate, a few D steps per G step, and regularising D.

6.3A deeper cause: JS divergence goes flat

Chapter 3 showed an ideal D turns the game into minimising JS divergence. That has a problem when the real and fake distributions do not overlap, which is the usual situation early on (images live on thin manifolds, Chapter 2). Take the extreme: real data is a spike at position 0 and the generator's spike is at $\theta$:

import numpy as np
grid = np.arange(0, 21)                                             # positions 0..20
def point(pos):
    p = np.zeros(21); p[pos] = 1.0; return p
def js(p, q):                                                       # Jensen-Shannon divergence in nats
    m = (p + q) / 2
    kl = lambda a, b: float(np.sum(a[a > 0] * np.log(a[a > 0] / b[a > 0])))
    return 0.5 * kl(p, m) + 0.5 * kl(q, m)
def w1(p, q):                                                       # Wasserstein-1 on a line: area between the CDFs
    return float(np.sum(np.abs(np.cumsum(p) - np.cumsum(q))))
real = point(0)
print("real data is a spike at 0; the generator's spike sits at position theta")
print(" theta   JS divergence   Wasserstein-1")
for theta in (0, 1, 2, 5, 10, 20):
    fake = point(theta)
    print(f"  {theta:4d}   {js(real, fake):12.4f}   {w1(real, fake):12.1f}")
print("log 2 =", round(float(np.log(2)), 4), ": once the two do not overlap JS is a constant, so it gives G no direction to move; Wasserstein keeps shrinking")

real data is a spike at 0; the generator's spike sits at position theta
 theta   JS divergence   Wasserstein-1
     0         0.0000            0.0
     1         0.6931            1.0
     2         0.6931            2.0
     5         0.6931            5.0
    10         0.6931           10.0
    20         0.6931           20.0
log 2 = 0.6931 : once the two do not overlap JS is a constant, so it gives G no direction to move; Wasserstein keeps shrinking

For every $\theta\neq0$, JS is the same $\log2=0.693$: it cannot tell a generator that is one step away from one that is twenty steps away, so its gradient is zero and G learns nothing. The Wasserstein-1 (earth mover's) distance is simply how far, on average, the probability mass has to be carried: 1, 2, 5, 10, 20, a useful direction at every distance. The Wasserstein GAN (WGAN) trains a critic to estimate this distance (with a constraint that the critic be smooth, enforced by weight clipping or a gradient penalty), which gives G a meaningful gradient even when D is far ahead and makes training far less fragile. This is the main fix to know.

6.4Measuring quality: FID

Because the loss does not measure quality, GANs are judged by their samples. The standard number is the Fréchet Inception Distance (FID): run real and generated images through a pre-trained classifier, take the feature vectors, fit a Gaussian (mean $\mu$, covariance $\Sigma$) to each set, and compute the distance between the two Gaussians:

Lower is better, and, crucially, it compares sets, so it sees what per-image inspection cannot. In the demo the "features" of real images form two clusters:

import numpy as np
rng = np.random.default_rng(0)
def fid(a, b):                                       # Frechet distance between Gaussians fitted to two feature sets
    m1, m2 = a.mean(0), b.mean(0); S1, S2 = np.cov(a.T), np.cov(b.T)
    root = np.sum(np.sqrt(np.clip(np.linalg.eigvals(S1 @ S2).real, 0, None)))   # trace of sqrt(S1 S2)
    return float(np.sum((m1 - m2) ** 2) + np.trace(S1) + np.trace(S2) - 2 * root)

def two_clusters(n):                                 # "features" of real images: two kinds of image (two clusters)
    c = rng.integers(0, 2, n)
    return np.where(c[:, None] == 0, [-3.0, 0.0], [3.0, 0.0]) + rng.normal(0, 0.5, (n, 2))
real = two_clusters(5000)
good = two_clusters(5000)                            # a generator that covers both kinds
collapsed = rng.normal([3.0, 0.0], 0.5, (5000, 2))   # a generator that only ever draws the right-hand kind
blurry = real + rng.normal(0, 1.5, real.shape)       # covers both kinds but every sample is noisy
print("FID (lower is better)")
print("  good generator, both kinds       :", round(fid(real, good), 3))
print("  mode-collapsed, one kind only    :", round(fid(real, collapsed), 3), " <- each sample is perfectly realistic, yet FID is large")
print("  covers both but noisy samples    :", round(fid(real, blurry), 3))

FID (lower is better)
  good generator, both kinds       : 0.006
  mode-collapsed, one kind only    : 15.372  <- each sample is perfectly realistic, yet FID is large
  covers both but noisy samples    : 1.352

A generator covering both clusters scores 0.006. A mode-collapsed one, whose every sample is a perfectly good member of one cluster, scores 15.4, thousands of times worse, because the set misses half the data. (The notes' line "FID is high even if each image looks real" is exactly this.) A generator that covers both but adds noise to everything sits in between (1.35). FID needs thousands of samples and a fixed reference network to be comparable between papers, and, like any single number, it has blind spots.

6.5Chapter summary

• Mode collapse: G converges to one safe mode. Measured: every toy run first put 0% or 100% of samples on one mode. Fixes: minibatch discrimination, feature matching, WGAN.
• Non-convergence: simultaneous gradient play circles and drifts out (distance 1.41 to 2.33).
• JS is flat at log 2 for disjoint distributions; Wasserstein keeps a gradient. FID compares feature sets: 0.006 good, 15.4 collapsed.

What is likely to be asked from Week 6, with 30-second answers, deeper explanations, a lecture-or-extra tag on each question, and code you can run.

Week 6 questions are about the GAN game: the objective, why the practical loss differs, what goes wrong, and how it is measured. Each answer starts with a 30-second version. Questions are tagged From the notes or Extra.

7.1Interview heat-map: what is most likely

7.2Generative modelling

learned: cat ~ N(45.8, 5.0),  dog ~ N(61.6, 7.7),  P(cat) = 0.4
  P(cat | length=45) = 0.911
  P(cat | length=52) = 0.507
  P(cat | length=55) = 0.213
  P(cat | length=60) = 0.018
decision boundary: generative 52.1 cm, discriminative 51.6 cm
new synthetic cats: [51.7 47.3 46.8 47.1 39. ]
log-density of a 46 cm cat: -2.52   of a 100 cm 'cat': -61.82 (anomaly detection)

images: 500 x 64 pixels, generated from only 2 latent numbers (cx, cy)
share of variance by PCA component: [0.343, 0.291, 0.116, 0.101, 0.066, 0.025, 0.023, 0.01]
cumulative, first 2 / 4 / 8 components: [0.635, 0.852, 0.975]
halfway in PIXEL space : max brightness 0.41 (two half-bright ghosts, not a blob in the middle)
halfway in LATENT space: max brightness 0.81 (one full blob, in the middle)

7.3The GAN game

at initialisation D(x) = 0.5 everywhere:
  D loss = 1.3863  = 2 log 2 = 1.3863
  G loss = 0.6931  = log 2   = 0.6931

D*(x) = [0.25, 0.8, 0.6, 0.6, 0.143]  (high where real data is more likely than fake)
V(D*, G)              = -1.1234
-log 4 + 2 * JS(p_data || p_g) = -1.1234   (JS = 0.1314)  -> they match
a generator that matches the data exactly:
  D* = [0.5, 0.5, 0.5, 0.5, 0.5]   V = -1.3863   = -log 4 = -1.3863

D is confident that a sample is fake when its logit a is very negative. How strong is G's learning signal?
  logit a   D(G(z))   gradient of  log(1-D)  [original]   gradient of  -log D  [non-saturating]
    -6.0    0.0025                 -0.0025                           -0.9975
    -4.0    0.0180                 -0.0180                           -0.9820
    -2.0    0.1192                 -0.1192                           -0.8808
    +0.0    0.5000                 -0.5000                           -0.5000
    +2.0    0.8808                 -0.8808                           -0.1192
both push D(G(z)) up (negative gradient = go up in logit), but when D wins the original loss is almost silent

fake = G(z)
loss_D = bce(D(real), ones) + bce(D(fake.detach()), zeros)   # detach: no gradient flows into G
optD.zero_grad(); loss_D.backward(); optD.step()

loss_G = bce(D(fake), ones)                                    # NOT detached: the gradient must reach G
optG.zero_grad(); loss_G.backward(); optG.step()             # only G steps

7.4Architectures

generator shapes:
   ConvTranspose2d  (2, 512, 4, 4)
   ConvTranspose2d  (2, 256, 8, 8)
   ConvTranspose2d  (2, 128, 16, 16)
   ConvTranspose2d  (2, 64, 32, 32)
   ConvTranspose2d  (2, 3, 64, 64)
   Tanh             (2, 3, 64, 64)

checked 132 combinations against torch: 0 mismatches
the DCGAN doubling layer (kernel 4, stride 2, padding 1): ['4->8', '8->16', '16->32', '32->64']
the first layer (kernel 4, stride 1, padding 0) turns a 1x1 noise vector into: 1->4

content feature maps, per channel (mean, std): [(0.09, 1.05), (10.06, 5.21), (-2.01, 0.16)]
after AdaIN, per channel (mean, std)           : [(1.0, 2.0), (-1.0, 0.5), (-0.0, 1.0)]
requested style (shift, scale)                 : [(1.0, 2.0), (-1.0, 0.5), (0.0, 1.0)]
the content's own statistics are erased and replaced by the style's; its spatial pattern is kept

7.5Failures and evaluation

real data: 50% of samples near -3 and 50% near +3, so the fraction of samples above 0 should be 0.50
fraction of generated samples above 0 (0.00 = all in the left mode, 1.00 = all in the right mode):
  seed     250    500   1000   1500   2000   3000      std at 3000
     0    0.00   0.00   0.00   0.47   0.47   0.46      3.01  (real std 3.04)
     3    1.00   1.00   1.00   1.00   1.00   0.52      3.02  (real std 3.04)
     4    1.00   1.00   1.00   1.00   1.00   0.49      3.11  (real std 3.04)

step   x        y        distance from the equilibrium
   0   +1.000   +1.000     1.414
  10   -0.312   +1.453     1.486
  25   -1.587   -0.216     1.602
  50   +1.579   -0.892     1.814
 100   -0.560   -2.257     2.326
each step multiplies the distance by sqrt(1 + eta^2) = 1.00499  (>1): it spirals OUT, never in

real data is a spike at 0; the generator's spike sits at position theta
 theta   JS divergence   Wasserstein-1
     0         0.0000            0.0
     1         0.6931            1.0
     2         0.6931            2.0
     5         0.6931            5.0
    10         0.6931           10.0
    20         0.6931           20.0
log 2 = 0.6931 : once the two do not overlap JS is a constant, so it gives G no direction to move; Wasserstein keeps shrinking

FID (lower is better)
  good generator, both kinds       : 0.006
  mode-collapsed, one kind only    : 15.372  <- each sample is perfectly realistic, yet FID is large
  covers both but noisy samples    : 1.352

Week 6 on one page: chapter glance, the GAN formulas, the pairs to tell apart, a failure-and-fix table, a checklist and corrections to the source notes.

8.1Week 6 at a glance

8.2Formulas

8.3Key terms

8.4The pairs you must tell apart

8.5Failure and fix table

8.6Last-minute checklist

8.7Corrections and additions to the source notes