Deep Learning,
Week 11

Week 6 introduced GANs. This week a different family takes over the image-generation world: diffusion models, the engine behind Stable Diffusion, DALL-E and Imagen. To see why, put the three families side by side first.

1.1VAE and GAN: two philosophies

• VAE (variational autoencoder): an encoder compresses an image to a latent $z$, a decoder rebuilds the image from $z$; training maximises the ELBO (evidence lower bound: a reconstruction term plus a term keeping the latent close to a Gaussian). It gives a smooth, structured latent space you can interpolate in, and trains stably, but its outputs are often blurry.
• GAN (Week 6): a generator against a discriminator. Sharp images and fast sampling, but unstable training, mode collapse and no structured latent space.

Why are VAE outputs blurry? Not because of anything mysterious in the architecture, but because of the loss. A mean-squared-error reconstruction loss rewards the average of all the sharp images that could fit a description, and an average of sharp images is a blur. A minimal version of the effect: one input, two equally likely correct outputs $+1$ and $-1$.

import numpy as np
rng = np.random.default_rng(0)
# one input, two equally likely correct outputs: +1 or -1 (think of two different sharp images that both fit the same description)
n = 100000
y = rng.choice([-1.0, 1.0], n)
blur = y.mean()                                           # the single value that minimises mean squared error is the MEAN
coin = rng.choice([-1.0, 1.0], n)                         # a sampler that commits to +1 or -1 at random
print(f"the answer that minimises squared error: {blur:+.3f}, an 'average' of the two correct answers")
print(f"  squared error of the average         : {((y - blur) ** 2).mean():.3f}")
print(f"  squared error of committing randomly: {((y - coin) ** 2).mean():.3f}   <- squared error PREFERS the average")
print(f"distance from the average to the nearest real answer: {min(abs(blur - 1), abs(blur + 1)):.3f} (it is not a valid answer at all)")
print("so a decoder trained with MSE (a plain VAE) learns the mean of everything that could fit, which in an image is a blur;")
print("a GAN or a diffusion model must commit to ONE sharp answer instead")

the answer that minimises squared error: -0.001, an 'average' of the two correct answers
  squared error of the average         : 1.000
  squared error of committing randomly: 2.002   <- squared error PREFERS the average
distance from the average to the nearest real answer: 0.999 (it is not a valid answer at all)
so a decoder trained with MSE (a plain VAE) learns the mean of everything that could fit, which in an image is a blur;
a GAN or a diffusion model must commit to ONE sharp answer instead

The best answer under squared error is the average, 0, which is not a valid answer at all (it is 1.0 from both real ones); and its error (1.0) even beats committing to a random sharp answer (2.0). That is exactly how an MSE-trained decoder ends up blurring: it is doing what it was asked to do. A generative sampler must instead commit to one sharp answer, which is what GANs and diffusion models do.

1.2The generative trilemma

A useful (and simplified) summary of why diffusion models matter: a generator would ideally give high-quality samples, cover all the modes of the data (diversity) and sample fast. GANs give quality and speed but not diversity (mode collapse); VAEs give diversity and speed but not quality (blur); diffusion models give quality and diversity but are slow, because generation takes many network calls. The rest of the week is largely about making that third wish come true (DDIM, latent diffusion) without losing the first two.

1.3Measuring generated images

• FID (Fréchet Inception Distance, Week 6): compare Gaussians fitted to Inception features of real and generated images, $\|\mu_r-\mu_g\|^2+\mathrm{Tr}(\Sigma_r+\Sigma_g-2(\Sigma_r\Sigma_g)^{1/2})$. Lower is better (0 means identical statistics); sensitive to both quality and diversity.
• Inception Score (IS): a pre-trained classifier should be confident on each generated image (low-entropy $P(y|x)$: sharp, recognisable) while the images together span many classes (high-entropy $P(y)$: diverse). $\mathrm{IS}=\exp\big(\mathbb{E}_x\,\mathrm{KL}(P(y|x)\,\|\,P(y))\big)$. Higher is better. It never looks at real images.
• CLIP score (for text-to-image): the cosine similarity between the CLIP embedding of the image and of the prompt. Higher is better: the image matches the text.

import numpy as np
def inception_score(p_yx):                                   # p_yx: (n images, n classes) = the classifier's P(y | x)
    p_y = p_yx.mean(0, keepdims=True)                        # the marginal P(y)
    kl = (p_yx * (np.log(p_yx + 1e-12) - np.log(p_y + 1e-12))).sum(1)
    return float(np.exp(kl.mean()))
n, k = 1000, 10
rng = np.random.default_rng(0)
sharp_diverse = np.full((n, k), 0.0005); sharp_diverse[np.arange(n), rng.integers(0, k, n)] = 0.9955      # confident, all 10 classes used
sharp_collapsed = np.full((n, k), 0.0005); sharp_collapsed[:, 3] = 0.9955                                   # confident, always class 3
blurry = np.full((n, k), 1.0 / k)                                                                          # classifier cannot decide
print("Inception Score = exp( mean KL( P(y|x) || P(y) ) )   (1 is the worst, the number of classes is the best)")
print(f"  sharp images, all 10 classes   : {inception_score(sharp_diverse):6.2f}   <- realistic AND diverse")
print(f"  sharp images, one class only   : {inception_score(sharp_collapsed):6.2f}   <- mode collapse is caught")
print(f"  blurry images (P(y|x) uniform) : {inception_score(blurry):6.2f}   <- unrealistic is caught")
print("IS never looks at the real images, which is its limitation; FID (Week 6) compares generated features with real features")

Inception Score = exp( mean KL( P(y|x) || P(y) ) )   (1 is the worst, the number of classes is the best)
  sharp images, all 10 classes   :   9.58   <- realistic AND diverse
  sharp images, one class only   :   1.00   <- mode collapse is caught
  blurry images (P(y|x) uniform) :   1.00   <- unrealistic is caught
IS never looks at the real images, which is its limitation; FID (Week 6) compares generated features with real features

IS behaves as described: sharp images over all 10 classes score 9.58 (the maximum is 10), the same sharp images all from one class score 1.00 (mode collapse is punished), and a classifier that cannot decide (a blur) also scores 1.00. Note the direction, a classic slip: FID lower is better, IS and CLIP score higher is better.

1.4Chapter summary

• VAE: encoder + decoder, ELBO, smooth latent, blurry because MSE regresses to the mean (average of $\pm1$ is 0). GAN: sharp, but collapses.
• Trilemma: quality, diversity, speed; diffusion gives quality + diversity at the cost of many steps.
• FID lower is better; IS and CLIP score higher is better (IS: 9.58 diverse, 1.00 collapsed).

The idea behind diffusion models is borrowed from physics. Drop ink into water: it spreads until the water is uniformly coloured. That forward process, order into disorder, is easy and needs no learning. The reverse, un-mixing the ink, is impossible to do by hand, but if you take the forward process in tiny steps, undoing one tiny step is a simple, learnable job. A diffusion model destroys an image by adding noise in many small steps, then learns to reverse each one.

2.1One step of noise

$\beta_t$ is a small number (the noise schedule, for example growing from $10^{-4}$ to 0.02 over $T=1000$ steps). Each step shrinks the previous image slightly ($\sqrt{1-\beta_t}<1$) and adds a little Gaussian noise. The factors are chosen so that the variance stays at 1 (a "variance-preserving" process): if $x_{t-1}$ has variance 1, so does $x_t$. After $T$ steps nothing of the image is left and $x_T$ is pure Gaussian noise $\mathcal{N}(0,I)$. There is nothing to learn here: the forward process is fixed.

2.2The closed form: jump to any step

Training needs a noisy image at a random step $t$, and iterating $t$ small steps would be slow. Fortunately the repeated steps collapse into one. Define $\alpha_t=1-\beta_t$ and $\bar\alpha_t=\prod_{s=1}^{t}\alpha_s$. A sum of independent Gaussians is a Gaussian, so

The second form is the reparameterisation trick: instead of "sample $x_t$ from a distribution", write $x_t$ as a deterministic function of the data $x_0$ and an independent noise draw $\varepsilon$. That is what lets us produce $x_t$ for any $t$ in one line (and, in a VAE, lets gradients flow through the sampling). Check it numerically: run 500 small steps on 200,000 copies of a one-pixel image of value 1.5 and compare with one jump.

import numpy as np
T = 1000
betas = np.linspace(1e-4, 0.02, T)                 # the linear noise schedule of the original DDPM
alphas = 1 - betas
alpha_bar = np.cumprod(alphas)                     # alpha_bar_t = product of (1 - beta_s) for s = 1..t
print("   t    alpha_bar   sqrt(alpha_bar)  sqrt(1-alpha_bar)   signal-to-noise  alpha_bar/(1-alpha_bar)")
for t in (1, 10, 100, 250, 500, 750, 1000):
    ab = alpha_bar[t - 1]
    print(f"{t:5d}   {ab:9.5f}   {np.sqrt(ab):13.3f}   {np.sqrt(1 - ab):15.3f}   {ab / (1 - ab):18.4f}")

# the closed form x_t = sqrt(ab) x_0 + sqrt(1-ab) eps against running the one-step process t times
rng = np.random.default_rng(0)
x0 = 1.5                                           # a one-pixel "image" with value 1.5
t, n = 500, 200000
x = np.full(n, x0)
for s in range(t):                                 # iterate: x_s = sqrt(1-beta_s) x_{s-1} + sqrt(beta_s) eps
    x = np.sqrt(1 - betas[s]) * x + np.sqrt(betas[s]) * rng.normal(size=n)
direct = np.sqrt(alpha_bar[t - 1]) * x0 + np.sqrt(1 - alpha_bar[t - 1]) * rng.normal(size=n)
print(f"\nafter {t} steps: iterated mean {x.mean():.4f}, std {x.std():.4f}   |   closed form mean {direct.mean():.4f}, std {direct.std():.4f}   (theory: mean {np.sqrt(alpha_bar[t - 1]) * x0:.4f}, std {np.sqrt(1 - alpha_bar[t - 1]):.4f})")
print("one jump to any t gives the same distribution as t small steps: training never has to iterate")

   t    alpha_bar   sqrt(alpha_bar)  sqrt(1-alpha_bar)   signal-to-noise  alpha_bar/(1-alpha_bar)
    1     0.99990           1.000             0.010            9999.0000
   10     0.99811           0.999             0.044             526.7615
  100     0.89702           0.947             0.321               8.7104
  250     0.52409           0.724             0.690               1.1012
  500     0.07859           0.280             0.960               0.0853
  750     0.00335           0.058             0.998               0.0034
 1000     0.00004           0.006             1.000               0.0000

after 500 steps: iterated mean 0.4226, std 0.9606   |   closed form mean 0.4245, std 0.9616   (theory: mean 0.4205, std 0.9599)
one jump to any t gives the same distribution as t small steps: training never has to iterate

After 500 small steps the mean and standard deviation (0.4226 and 0.9606) agree with one jump (0.4245 and 0.9616) and with the theory (0.4205 and 0.9599). The table shows how the image dissolves for the linear schedule: $\bar\alpha_t$ falls from 0.9999 at $t=1$ to 0.897 at $t=100$, 0.524 at 250, 0.079 at 500, 0.003 at 750 and $4\times10^{-5}$ at 1000, where the image coefficient is $\sqrt{\bar\alpha}=0.006$ and the noise coefficient is 1.000. The signal-to-noise ratio $\bar\alpha/(1-\bar\alpha)$ crosses 1 around $t=250$.

2.3Reading the notes' example correctly

The notes' worked example takes $\bar\alpha_t=0.1$ and writes $x_t=0.316\,x_0+0.949\,\varepsilon$, then says "only 31.6% of the original image remains, 94.9% noise". The algebra is right; the sentence is not.

import numpy as np
ab = 0.1                                           # the notes' example: alpha_bar = 0.1 at t = 500
print("x_t = sqrt(0.1) x_0 + sqrt(0.9) eps =", round(np.sqrt(ab), 3), "x_0 +", round(np.sqrt(1 - ab), 3), "eps")
rng = np.random.default_rng(0)
x0 = rng.normal(0, 1, 100000); eps = rng.normal(0, 1, 100000)
xt = np.sqrt(ab) * x0 + np.sqrt(1 - ab) * eps
print("variance of x_t:", round(xt.var(), 3), "(x_0 and eps each have variance 1, so the variance stays 1: the process is variance-preserving)")
print("share of the VARIANCE that is signal:", round(ab, 3), "  share that is noise:", round(1 - ab, 3))
print("share of the COEFFICIENT that is signal:", round(np.sqrt(ab), 3), "(this is the 0.316; it is a multiplier, not '31.6% of the image')")
print("correlation between x_t and x_0:", round(float(np.corrcoef(xt, x0)[0, 1]), 3), " = sqrt(alpha_bar)")
# for the real linear schedule, what is alpha_bar at t = 500?
alpha_bar = np.cumprod(1 - np.linspace(1e-4, 0.02, 1000))
print("for the linear schedule 1e-4 .. 0.02 the true value at t = 500 is", round(float(alpha_bar[499]), 3), "(the notes' 0.1 is an illustration)")

x_t = sqrt(0.1) x_0 + sqrt(0.9) eps = 0.316 x_0 + 0.949 eps
variance of x_t: 1.005 (x_0 and eps each have variance 1, so the variance stays 1: the process is variance-preserving)
share of the VARIANCE that is signal: 0.1   share that is noise: 0.9
share of the COEFFICIENT that is signal: 0.316 (this is the 0.316; it is a multiplier, not '31.6% of the image')
correlation between x_t and x_0: 0.317  = sqrt(alpha_bar)
for the linear schedule 1e-4 .. 0.02 the true value at t = 500 is 0.079 (the notes' 0.1 is an illustration)

0.316 and 0.949 are multipliers (they do not add to 100%: $0.316+0.949=1.265$). What adds to 1 are their squares: 10% of the variance is signal and 90% is noise, and the result keeps variance 1.005, confirming the variance-preserving property. Equivalently, $x_t$ and $x_0$ have correlation $\sqrt{\bar\alpha}=0.317$. Also note that the example's $\bar\alpha_{500}=0.1$ is illustrative: for the linear schedule it is 0.079.

2.4Noise schedules

import numpy as np
T = 1000
t = np.arange(1, T + 1)
linear = np.cumprod(1 - np.linspace(1e-4, 0.02, T))
s = 0.008
f = lambda u: np.cos((u / T + s) / (1 + s) * np.pi / 2) ** 2           # the cosine schedule of Nichol and Dhariwal
cosine = f(t) / f(0)
print("alpha_bar (how much of the original signal's variance is left)")
print("     t      linear    cosine")
for k in (50, 100, 250, 500, 750, 900, 1000):
    print(f"{k:6d}   {linear[k - 1]:8.4f}  {cosine[k - 1]:8.4f}")
print("linear destroys the image early: by t = 500 only", round(float(linear[499]), 3), "of the signal is left, so the last half of the steps work on almost pure noise")
print("cosine destroys it more gradually: still", round(float(cosine[499]), 3), "left at t = 500")
print("first step alpha_bar:  linear", round(float(linear[0]), 5), " cosine", round(float(cosine[0]), 5))

alpha_bar (how much of the original signal's variance is left)
     t      linear    cosine
    50     0.9710    0.9920
   100     0.8970    0.9721
   250     0.5241    0.8470
   500     0.0786    0.4938
   750     0.0034    0.1443
   900     0.0003    0.0241
  1000     0.0000    0.0000
linear destroys the image early: by t = 500 only 0.079 of the signal is left, so the last half of the steps work on almost pure noise
cosine destroys it more gradually: still 0.494 left at t = 500
first step alpha_bar:  linear 0.9999  cosine 0.99996

The schedule decides how fast the image is destroyed. The linear schedule ($\beta$ rising linearly) leaves only 7.9% of the signal at $t=500$, so the second half of the steps work on near-pure noise, which wastes capacity. The cosine schedule (Nichol and Dhariwal) is smoother: 49% is still left at $t=500$ and it reaches 0 only at the end, which is reported to give better samples. A quadratic schedule grows $\beta$ as the square of $t$, adding noise aggressively early. All three share the same closed form; only $\bar\alpha_t$ differs.

2.5Chapter summary

• Forward step $x_t=\sqrt{1-\beta_t}x_{t-1}+\sqrt{\beta_t}\varepsilon$; $x_T\approx\mathcal{N}(0,I)$. Fixed, no training.
• Closed form $x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\varepsilon$ (reparameterisation): jump to any $t$; verified against 500 iterated steps. Linear $\bar\alpha_{500}=0.079$, cosine 0.494.
• 0.316 and 0.949 are coefficients, not percentages: the variance is 10% signal and 90% noise.

The reverse process is where learning happens. Starting from pure noise $x_T$, we want a network that walks back to a clean image, one small step at a time, by modelling $p_\theta(x_{t-1}\mid x_t)$. Because each forward step was tiny, the true reverse step is also (very nearly) Gaussian, so the network only has to predict its mean (the variance can be fixed).

3.1What is the right reverse step?

If we knew the clean image $x_0$, the exact reverse step would be a Gaussian with a known mean (the posterior):

The network does not know $x_0$, but it does not need to: since $x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\varepsilon$, knowing $x_t$ and the noise $\varepsilon$ tells you $x_0$. Substituting, the same mean can be written purely with the noise:

import numpy as np
rng = np.random.default_rng(0)
T = 1000
betas = np.linspace(1e-4, 0.02, T); alphas = 1 - betas; abar = np.cumprod(alphas)
t = 400                                                       # 0-based index of the step
x0 = rng.normal(size=5); eps = rng.normal(size=5)
xt = np.sqrt(abar[t]) * x0 + np.sqrt(1 - abar[t]) * eps       # a noisy point made with a KNOWN noise eps

# (a) the true posterior mean q(x_{t-1} | x_t, x_0), the formula in the notes
mu_post = (np.sqrt(abar[t - 1]) * betas[t] / (1 - abar[t])) * x0 + (np.sqrt(alphas[t]) * (1 - abar[t - 1]) / (1 - abar[t])) * xt
# (b) the same mean written with the noise: this is what the sampler computes once eps_theta predicts eps
mu_eps = (xt - betas[t] / np.sqrt(1 - abar[t]) * eps) / np.sqrt(alphas[t])
print("posterior mean (notes' formula)       :", mu_post.round(4))
print("mean written through the noise epsilon:", mu_eps.round(4))
print("identical:", np.allclose(mu_post, mu_eps), " -> predicting the NOISE is enough to take a denoising step")
print("posterior variance beta_tilde =", f"{betas[t] * (1 - abar[t - 1]) / (1 - abar[t]):.6f}", " (beta_t =", f"{betas[t]:.6f})")

posterior mean (notes' formula)       : [ 0.3783  1.1056  1.1283 -0.5818 -1.3662]
mean written through the noise epsilon: [ 0.3783  1.1056  1.1283 -0.5818 -1.3662]
identical: True  -> predicting the NOISE is enough to take a denoising step
posterior variance beta_tilde = 0.008052  (beta_t = 0.008068)

The two expressions agree to every digit (True). This is the heart of DDPM: a network that predicts the noise $\varepsilon$ in $x_t$ is enough to take a denoising step. So we train $\varepsilon_\theta(x_t,t)$ to predict the noise, and plug its output into the mean above.

3.2The training objective

• Sample a clean image $x_0$ from the data.
• Sample a random step $t\sim\mathrm{Uniform}(1,T)$ and a noise $\varepsilon\sim\mathcal{N}(0,I)$.
• Make the noisy image in one jump with the closed form.
• Predict the noise with the network and minimise the mean squared error between the true and predicted noise.

Three lines of code: pick $x_0,t,\varepsilon$; add noise in one jump; minimise $\|\varepsilon-\varepsilon_\theta\|^2$. The network is trained as a plain regression, with none of the instability of a GAN. Why predict the noise rather than the clean image? (1) It is equivalent, as just shown, to predicting the denoising mean. (2) The target $\varepsilon$ always has the same, well-behaved scale (mean 0, variance 1) at every $t$, unlike the clean image, whose best estimate at high $t$ is hard to scale. (3) It matches denoising score matching, a well-studied objective. The notes' mistake-list is right: DDPM predicts the noise, not the image.

3.3The network: a U-Net with the step as an extra input

For images the noise predictor is a U-Net (Week 4): the encoder-decoder with skip connections is a natural fit because the output (a noise image) has the same size as the input and needs fine spatial detail. One new problem: the same network must work for every noise level, so it has to be told the step $t$. Feeding the bare integer is poor, because "step 10" and "step 11" should be treated as similar. The fix is borrowed from Chapter 1 of Week 10: a sinusoidal time embedding, the same formula as the Transformer's positional encoding, passed through a small MLP and added to the feature maps at every resolution of the U-Net.

import math, torch
def time_embedding(t, d=64):
    freq = torch.exp(-math.log(10000) * torch.arange(d // 2) / (d // 2))
    a = t[:, None].float() * freq[None]
    return torch.cat([a.sin(), a.cos()], 1)
t = torch.tensor([10, 11, 12, 20, 50, 500, 900])
E = time_embedding(t); E = E / E.norm(dim=1, keepdim=True)
print("cosine similarity of the embedding of step 10 with others (nearby steps should look alike):")
for j in range(1, len(t)):
    print(f"   step {t[0].item():3d} vs step {t[j].item():3d}: {(E[0] @ E[j]).item():+.3f}")
# how it enters a convolutional U-Net: project, then add to every spatial position of the feature maps
B, C, H, W = 4, 32, 16, 16
features = torch.randn(B, C, H, W)
proj = torch.nn.Sequential(torch.nn.Linear(64, 128), torch.nn.SiLU(), torch.nn.Linear(128, C))
emb = proj(time_embedding(torch.randint(0, 1000, (B,))))                      # (B, C): one vector per image in the batch
out = features + emb[:, :, None, None]                                          # broadcast over H and W
print("\nfeature maps", tuple(features.shape), "+ time embedding", tuple(emb.shape), "-> broadcast to", tuple(out.shape), "(added at every resolution of the U-Net)")
print("same sinusoidal idea as the Transformer's positional encoding, applied to the diffusion step instead of the token position")

cosine similarity of the embedding of step 10 with others (nearby steps should look alike):
   step  10 vs step  11: +0.966
   step  10 vs step  12: +0.884
   step  10 vs step  20: +0.658
   step  10 vs step  50: +0.481
   step  10 vs step 500: +0.216
   step  10 vs step 900: +0.155

feature maps (4, 32, 16, 16) + time embedding (4, 32) -> broadcast to (4, 32, 16, 16) (added at every resolution of the U-Net)
same sinusoidal idea as the Transformer's positional encoding, applied to the diffusion step instead of the token position

The embeddings of nearby steps are similar (cosine 0.966 for steps 10 and 11, 0.884 for 10 and 12) and fall off with distance (0.658 at 20, 0.216 at 500), and projecting it to the channel count and broadcasting over height and width adds one vector per image to every spatial position. This is the "time conditioning" of the notes.

3.4Chapter summary

• The reverse step is a Gaussian whose mean can be written with the noise: $\tilde\mu_t=\tfrac1{\sqrt{\alpha_t}}(x_t-\tfrac{\beta_t}{\sqrt{1-\bar\alpha_t}}\varepsilon)$ (verified identical to the posterior formula).
• Train $\varepsilon_\theta(x_t,t)$ to predict the noise: $L=\mathbb{E}\|\varepsilon-\varepsilon_\theta(x_t,t)\|^2$; a plain regression, stable. Predict the noise, not the image.
• U-Net backbone; the step $t$ enters through a sinusoidal embedding (neighbouring steps have cosine 0.97) added at every resolution.

With a trained noise predictor, generation starts from pure noise $x_T\sim\mathcal{N}(0,I)$ and removes noise step by step down to $x_0$. The way the steps are taken gives two samplers with very different properties. To see everything run, here is a complete toy diffusion model: the data are 8 tight Gaussian blobs on a circle (a distribution that a model must cover completely), the network is a small MLP with the sinusoidal time embedding, and the schedule is scaled to $T=100$ steps. The first lines of output are the training loss (Chapter 3's three-line recipe); the rest are the samplers of this chapter.

import math, torch, torch.nn as nn
torch.set_num_threads(1); torch.manual_seed(0)
# data: 8 tight Gaussian blobs on a circle of radius 2 (a distribution a model must cover completely)
CENTRES = torch.stack([2 * torch.cos(torch.arange(8) * math.pi / 4), 2 * torch.sin(torch.arange(8) * math.pi / 4)], 1)
def data(n):
    return CENTRES[torch.randint(0, 8, (n,))] + 0.1 * torch.randn(n, 2)

T = 100
betas = torch.linspace(1e-3, 0.2, T); alphas = 1 - betas; abar = torch.cumprod(alphas, 0)     # a schedule scaled for 100 steps

def time_embedding(t, d=32):                                      # sinusoidal embedding of the step index, as in the Transformer
    freq = torch.exp(-math.log(10000) * torch.arange(d // 2) / (d // 2))
    a = t[:, None].float() * freq[None]
    return torch.cat([a.sin(), a.cos()], 1)

class NoisePredictor(nn.Module):                                   # eps_theta(x_t, t)
    def __init__(self, h=128):
        super().__init__()
        self.t = nn.Sequential(nn.Linear(32, h), nn.SiLU()); self.x = nn.Linear(2, h)
        self.body = nn.Sequential(nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 2))
    def forward(self, x, t):
        return self.body(self.x(x) + self.t(time_embedding(t)))

net = NoisePredictor(); opt = torch.optim.Adam(net.parameters(), lr=2e-3)
for step in range(1, 6001):                                        # TRAINING: the three lines from the notes
    x0 = data(512); t = torch.randint(0, T, (512,)); eps = torch.randn_like(x0)       # 1. pick data and a random step; 2. draw noise
    ab = abar[t][:, None]; xt = ab.sqrt() * x0 + (1 - ab).sqrt() * eps               #    add noise in ONE jump (closed form)
    loss = ((net(xt, t) - eps) ** 2).mean()                                          # 3. predict the noise, mean squared error
    opt.zero_grad(); loss.backward(); opt.step()
    if step in (1, 500, 6000): print(f"step {step:4d}: loss {loss.item():.3f}   (predicting zero noise would give about 1.0)")

@torch.no_grad()
def ddpm(xT, seed):                                                # ancestral sampling: T network calls, fresh noise at every step
    g = torch.Generator().manual_seed(seed); x = xT.clone()
    for t in reversed(range(T)):
        eps = net(x, torch.full((len(x),), t))
        x = (x - (1 - alphas[t]) / (1 - abar[t]).sqrt() * eps) / alphas[t].sqrt()
        if t > 0: x = x + (betas[t] * (1 - abar[t - 1]) / (1 - abar[t])).sqrt() * torch.randn(x.shape, generator=g)
    return x

@torch.no_grad()
def ddim(xT, steps):                                               # deterministic, can use a SUBSET of the steps
    x = xT.clone(); ts = torch.linspace(T - 1, 0, steps).long().tolist()
    for i, t in enumerate(ts):
        eps = net(x, torch.full((len(x),), t)); x0_hat = (x - (1 - abar[t]).sqrt() * eps) / abar[t].sqrt()
        x = x0_hat if i + 1 == len(ts) else abar[ts[i + 1]].sqrt() * x0_hat + (1 - abar[ts[i + 1]]).sqrt() * eps
    return x

def quality(x):
    d = torch.cdist(x, CENTRES); near = d.min(1)
    ok = near.values < 0.4                                          # within 4 standard deviations of a blob centre
    cover = torch.bincount(near.indices[ok], minlength=8).float() / len(x)
    return ok.float().mean().item(), cover.min().item(), cover.max().item()

xT = torch.randn(2000, 2, generator=torch.Generator().manual_seed(1))
print(f"\nreal data: {quality(data(2000))[0]:.3f} of points on a blob (each blob should get 0.125)")
print("sampler        network calls   points on a blob   smallest / largest blob share")
for name, calls, x in (("DDPM", 100, ddpm(xT, 0)), ("DDIM 50 steps", 50, ddim(xT, 50)), ("DDIM 20 steps", 20, ddim(xT, 20)), ("DDIM 10 steps", 10, ddim(xT, 10)), ("DDIM 5 steps", 5, ddim(xT, 5)), ("DDIM 2 steps", 2, ddim(xT, 2))):
    q = quality(x); print(f"{name:14s} {calls:8d}        {q[0]:9.3f}          {q[1]:.2f} / {q[2]:.2f}")

a = ddpm(xT[:5], 0); b = ddpm(xT[:5], 1)
print("\nsame starting noise x_T:")
print("  DDPM run twice with different step noise: mean difference between the two results =", round((a - b).abs().mean().item(), 3), "(stochastic)")
print("  DDIM run twice:                          mean difference between the two results =", round((ddim(xT[:5], 20) - ddim(xT[:5], 20)).abs().mean().item(), 3), "(deterministic)")

@torch.no_grad()
def invert(x0, steps):                                             # DDIM inversion: run the deterministic process FORWARD, data -> noise
    x = x0.clone(); ts = torch.linspace(0, T - 1, steps).long().tolist()
    for i in range(len(ts) - 1):
        t, tn = ts[i], ts[i + 1]; eps = net(x, torch.full((len(x),), t))
        x0_hat = (x - (1 - abar[t]).sqrt() * eps) / abar[t].sqrt()
        x = abar[tn].sqrt() * x0_hat + (1 - abar[tn]).sqrt() * eps
    return x
real = data(1000); z = invert(real, 50); back = ddim(z, 50)
print(f"\nDDIM inversion of 1000 real points: reconstruction error {(back - real).norm(dim=1).mean().item():.3f} (blob width is 0.1*sqrt(2)=0.14); the recovered noise has mean {z.mean().item():+.2f}, std {z.std().item():.2f} (target 0, 1)")
nudged = ddim(z[:200] + 0.5 * torch.randn(200, 2), 50)
print(f"nudging the recovered noise by 0.5 and decoding again: the point stays on a blob in {quality(nudged)[0]:.2f} of cases")

step    1: loss 0.985   (predicting zero noise would give about 1.0)
step  500: loss 0.295   (predicting zero noise would give about 1.0)
step 6000: loss 0.220   (predicting zero noise would give about 1.0)

real data: 1.000 of points on a blob (each blob should get 0.125)
sampler        network calls   points on a blob   smallest / largest blob share
DDPM                100            0.985          0.10 / 0.14
DDIM 50 steps        50            0.956          0.08 / 0.19
DDIM 20 steps        20            0.915          0.06 / 0.20
DDIM 10 steps        10            0.762          0.04 / 0.19
DDIM 5 steps          5            0.273          0.00 / 0.15
DDIM 2 steps          2            0.014          0.00 / 0.01

same starting noise x_T:
  DDPM run twice with different step noise: mean difference between the two results = 2.585 (stochastic)
  DDIM run twice:                          mean difference between the two results = 0.0 (deterministic)

DDIM inversion of 1000 real points: reconstruction error 0.064 (blob width is 0.1*sqrt(2)=0.14); the recovered noise has mean +0.02, std 1.19 (target 0, 1)
nudging the recovered noise by 0.5 and decoding again: the point stays on a blob in 0.95 of cases

4.1DDPM: ancestral sampling

The original sampler repeats the reverse step $x_t\to x_{t-1}$ for $t=T,\dots,1$, each time computing the mean from the predicted noise and adding fresh random noise (of variance $\tilde\beta_t$). It is stochastic (two runs from the same $x_T$ give different images) and Markovian: each step depends only on the previous one, so you cannot skip steps; generation costs $T$ network calls (1,000 for the original model, about a minute per image). The toy table's first row: 100 calls put 98.5% of the generated points on a real blob, with every blob getting between 10% and 14% of them (ideal: 12.5%).

4.2DDIM: deterministic and fast

DDIM (Denoising Diffusion Implicit Models, Song et al., 2020) observes that the training objective does not depend on the forward process being a Markov chain: many non-Markovian processes share the same marginals $q(x_t\mid x_0)$ and so the same trained network. Choosing the deterministic one gives a sampler with no random noise added at inference: the predicted noise gives an estimate of the clean image, $\hat x_0=(x_t-\sqrt{1-\bar\alpha_t}\,\varepsilon_\theta)/\sqrt{\bar\alpha_t}$, and the next point is placed on the path to it, $x_{t_{\text{next}}}=\sqrt{\bar\alpha_{t_{\text{next}}}}\,\hat x_0+\sqrt{1-\bar\alpha_{t_{\text{next}}}}\,\varepsilon_\theta$. Because nothing depends on the immediately preceding step only, you can jump through a subset of the steps (1000, 980, 960, ... gives 50 steps).

Read the toy table. With the same network and the same starting noise: DDIM with 50 calls still puts 95.6% of points on a blob, 20 calls 91.5%, 10 calls 76.2%; below that it collapses (5 calls: 27%; 2 calls: 1.4%, the model has barely started). So DDIM buys a 10x to 20x speed-up for a modest quality loss, exactly the notes' "1000 $\to$ 50 steps, 20x faster", and the trade-off is gradual, not free. The determinism is measured too: from the same $x_T$, two DDPM runs differ by 2.585 on average, while two DDIM runs differ by 0.0: same noise in, same image out.

4.3DDIM inversion and editing

Because DDIM is deterministic, it can be run backwards: take a real image, apply the DDIM step in the direction of increasing noise, and obtain a noise $x_T$ that DDIM will decode back to that very image. This is DDIM inversion. The recovered $x_T$ "encodes" the image (its composition and content), so nudging it slightly and decoding gives an edited image, the basis of diffusion-based image editing and style transfer. On the toy model: inverting 1,000 real points and decoding them back reproduces them to an average error of 0.064 (the blob width is 0.14), the recovered noise has mean +0.02 and standard deviation 1.19 (the target is 0 and 1, so inversion is approximate here), and after nudging the recovered noise by 0.5 the decoded point still lands on a blob in 95% of cases: nearby noise decodes to a nearby, valid sample.

4.4Chapter summary

• DDPM: stochastic, Markovian, $T$ calls, cannot skip. Toy: 100 calls give 98.5% on-blob.
• DDIM: deterministic (measured difference 0.0 against 2.585), skips steps with the same network: 50, 20, 10 calls give 95.6%, 91.5%, 76.2%; 2 calls fail.
• DDIM inversion: image to noise $x_T$ and back (error 0.064); edit the noise, decode an edited image.

A plain diffusion model is unconditional: it produces a random sample from the data. Useful systems need control: "a cat", "a beautiful sunset". Guidance steers the denoising toward what we asked for.

5.1Classifier guidance

The first approach (Dhariwal and Nichol, 2021) uses a separate pre-trained classifier $p(y\mid x_t)$. The denoising direction is the score $\nabla\log p(x_t)$ (up to scale, the predicted noise); Bayes' rule gives the conditional score as the unconditional one plus the classifier's gradient, $\nabla\log p(x_t\mid y)=\nabla\log p(x_t)+\nabla\log p(y\mid x_t)$, so at each step the sample is pushed in the direction that makes the classifier more sure of the wanted class. The drawback: the classifier must be trained on noisy images of every noise level (an ordinary classifier has never seen them), and it only supports the fixed classes it was trained on, not free text.

5.2Classifier-free guidance (CFG)

Classifier-free guidance (Ho and Salimans, 2021) removes the classifier. Train one network $\varepsilon_\theta(x_t,t,c)$ that takes the condition $c$ (a class or a text embedding) as an extra input, and during training drop the condition a fraction of the time (10 to 20%), replacing $c$ by a "null" token $\varnothing$. One model thereby learns both the conditional prediction $\varepsilon_\theta(x_t,c)$ and the unconditional one $\varepsilon_\theta(x_t,\varnothing)$. At sampling time run both and extrapolate from the unconditional toward the conditional:

• $w=0$: the unconditional model; $w=1$: the plain conditional model; $w>1$: the difference $\varepsilon_c-\varepsilon_u$, "what the condition adds", is amplified. The notes' example: $w=7.5$ gives $\tilde\varepsilon=-6.5\,\varepsilon_u+7.5\,\varepsilon_c$ (check: $\varepsilon_u+7.5\varepsilon_c-7.5\varepsilon_u$).
• Typical text-to-image values are 7 to 15 (Stable Diffusion's default is 7.5). Higher $w$: more faithful to the prompt but less diverse, and, pushed far, over-saturated and artificial.
• Notation warning: the original CFG paper writes $(1+w)\varepsilon_c-w\,\varepsilon_u$, so its $w=0$ is the plain conditional model. The notes (and Stable Diffusion code) use the form above with $w=1$ conditional. The two differ by a shift of 1; quote the form you use.

5.3Guidance, measured on a toy problem

To see the effect without any images: two overlapping classes in 2-D, class 0 centred at $(-1,0)$ and class 1 at $(+1,0)$, each with standard deviation 1. One network is trained with the condition dropped 20% of the time (the null label is 2). Then sample class 1 with guidance scales from 0 to 8:

import math, torch, torch.nn as nn
torch.set_num_threads(1); torch.manual_seed(0)
# two OVERLAPPING classes: class 0 ~ N((-1, 0), I) and class 1 ~ N((+1, 0), I)
def data(n):
    c = torch.randint(0, 2, (n,)); mean = torch.stack([(2 * c - 1).float(), torch.zeros(n)], 1)
    return mean + torch.randn(n, 2), c
T = 100; betas = torch.linspace(1e-3, 0.2, T); alphas = 1 - betas; abar = torch.cumprod(alphas, 0)
def time_embedding(t, d=32):
    freq = torch.exp(-math.log(10000) * torch.arange(d // 2) / (d // 2)); a = t[:, None].float() * freq[None]
    return torch.cat([a.sin(), a.cos()], 1)
NULL = 2                                                              # label 2 = "no condition"
class Net(nn.Module):
    def __init__(self, h=128):
        super().__init__()
        self.t = nn.Sequential(nn.Linear(32, h), nn.SiLU()); self.x = nn.Linear(2, h); self.c = nn.Embedding(3, h)
        self.body = nn.Sequential(nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 2))
    def forward(self, x, t, c):
        return self.body(self.x(x) + self.t(time_embedding(t)) + self.c(c))
net = Net(); opt = torch.optim.Adam(net.parameters(), lr=2e-3)
for step in range(5000):
    x0, c = data(512)
    c = torch.where(torch.rand(512) < 0.2, torch.full_like(c, NULL), c)           # DROP the condition 20% of the time: ONE model learns both
    t = torch.randint(0, T, (512,)); eps = torch.randn_like(x0); ab = abar[t][:, None]
    loss = ((net(ab.sqrt() * x0 + (1 - ab).sqrt() * eps, t, c) - eps) ** 2).mean()
    opt.zero_grad(); loss.backward(); opt.step()

@torch.no_grad()
def sample(n, cls, w, seed=0):
    g = torch.Generator().manual_seed(seed); x = torch.randn(n, 2, generator=g)
    for t in reversed(range(T)):
        tt = torch.full((n,), t)
        eps_u = net(x, tt, torch.full((n,), NULL)); eps_c = net(x, tt, torch.full((n,), cls))
        eps = eps_u + w * (eps_c - eps_u)                                          # classifier-free guidance: eps_u + w (eps_c - eps_u)
        x = (x - (1 - alphas[t]) / (1 - abar[t]).sqrt() * eps) / alphas[t].sqrt()
        if t > 0: x = x + (betas[t] * (1 - abar[t - 1]) / (1 - abar[t])).sqrt() * torch.randn(n, 2, generator=g)
    return x

xr, cr = data(20000); real = xr[cr == 1]
print(f"real class-1 data: mean x {real[:, 0].mean():+.2f}, std x {real[:, 0].std():.2f}, fraction with x > 0: {(real[:, 0] > 0).float().mean():.3f}")
print("\nguidance scale w   mean x   std x   fraction x > 0   std y")
for w in (0, 1, 2, 4, 8):
    x = sample(3000, 1, w)
    print(f"{w:10d}      {x[:, 0].mean():+7.2f}  {x[:, 0].std():6.2f}   {(x[:, 0] > 0).float().mean():10.3f}      {x[:, 1].std():.2f}")
print("w = 0 is unconditional (both classes); w = 1 reproduces the real class-1 distribution; w > 1 pushes further and further into class 1 and the spread shrinks")

real class-1 data: mean x +0.99, std x 1.00, fraction with x > 0: 0.842

guidance scale w   mean x   std x   fraction x > 0   std y
         0        -0.11    1.39        0.465      1.05
         1        +0.95    0.97        0.832      0.99
         2        +1.43    0.75        0.978      0.95
         4        +1.78    0.60        0.999      0.88
         8        +1.89    0.47        1.000      0.75
w = 0 is unconditional (both classes); w = 1 reproduces the real class-1 distribution; w > 1 pushes further and further into class 1 and the spread shrinks

Four readings. (1) $w=0$ is the unconditional model: mean $-0.11$, standard deviation 1.39, because it draws from both classes at once. (2) $w=1$ reproduces the real class-1 data (mean 0.95 against 0.99, std 0.97 against 1.00, 83.2% of samples on the right side against 84.2%): the plain conditional model is a faithful sampler, and because the classes overlap about 17% of its samples honestly land on the "wrong" side. (3) Larger $w$ pushes samples into the class: 97.8% on the right side at $w=2$, 99.9% at $w=4$; the mean moves to 1.43, 1.78, 1.89, past the real mean of 0.99, an exaggeration of the class. (4) The cost is diversity: the spread falls from 0.97 to 0.47 at $w=8$, and even the unrelated $y$ direction shrinks (0.99 to 0.75). That is the "more faithful, less diverse" of the notes, seen in numbers: guidance trades diversity for fidelity.

5.4Chapter summary

• Classifier guidance adds $\nabla\log p(y|x_t)$ from a separate noisy-image classifier; CFG trains one model with the condition dropped 10 to 20% of the time.
• $\tilde\varepsilon=\varepsilon_u+w(\varepsilon_c-\varepsilon_u)$: $w=0$ unconditional, $w=1$ conditional, $w>1$ amplified (typical 7 to 15). Check the convention: the CFG paper uses $(1+w)\varepsilon_c-w\varepsilon_u$.
• Measured: $w=1$ matched the data (83.2% against 84.2%); $w=8$ put 100% in the class but cut the spread from 0.97 to 0.47: fidelity up, diversity down.

Everything so far works, but a $512\times512$ RGB image has $786{,}432$ numbers, and the U-Net must denoise all of them at every step, hundreds of times. Most of that effort is spent on tiny, perceptually invisible detail. Latent Diffusion Models (LDM) (Rombach et al., 2022) move the whole diffusion process to a much smaller space.

6.1Two stages

• Stage 1: train a VAE first to compress images into a small latent and decode them back (for example $512\times512\times3\to64\times64\times4$, height and width divided by 8). Then freeze it.
• Stage 2: train the diffusion model in the latent space (the U-Net denoises $64\times64\times4$ latents, with the same noise-prediction loss).
• Generation: sample $z_T\sim\mathcal{N}(0,I)$ of size $64\times64\times4$; denoise it to $z_0$ with the text condition; decode $z_0$ with the frozen VAE decoder into the image.

6.2How much does it save?

pixel = 512 * 512 * 3
latent = 64 * 64 * 4
print(f"pixel space : 512 x 512 x 3 = {pixel:,} numbers")
print(f"latent space: 64 x 64 x 4   = {latent:,} numbers   -> {pixel / latent:.0f}x fewer")
print(f"spatial positions: {512 * 512:,} against {64 * 64:,} = {(512 * 512) / (64 * 64):.0f}x fewer (the VAE shrinks height and width by 8)")
print()
print("what this does to the cost of the denoising network:")
print(f"  operations that grow with the number of positions (convolutions, FFN): about {(512 * 512) / (64 * 64):.0f}x cheaper")
print(f"  operations that grow with positions squared (self-attention over all positions): about {((512 * 512) / (64 * 64)) ** 2:.0f}x cheaper")
print("the notes' 2304x is the attention-only figure; a U-Net is mostly convolutions, so the realistic saving is a good deal smaller than 2304x")
print(f"memory of one float32 image: {pixel * 4 / 1e6:.1f} MB in pixel space, {latent * 4 / 1e3:.1f} KB as a latent")

pixel space : 512 x 512 x 3 = 786,432 numbers
latent space: 64 x 64 x 4   = 16,384 numbers   -> 48x fewer
spatial positions: 262,144 against 4,096 = 64x fewer (the VAE shrinks height and width by 8)

what this does to the cost of the denoising network:
  operations that grow with the number of positions (convolutions, FFN): about 64x cheaper
  operations that grow with positions squared (self-attention over all positions): about 4096x cheaper
the notes' 2304x is the attention-only figure; a U-Net is mostly convolutions, so the realistic saving is a good deal smaller than 2304x
memory of one float32 image: 3.1 MB in pixel space, 65.5 KB as a latent

The latent holds 16,384 numbers against 786,432: 48 times fewer (the notes' figure). The notes then claim a "2304 times faster" speed-up, $48^2$. Careful: height and width shrink by 8, so the number of spatial positions falls from 262,144 to 4,096, 64 times, and the channel count goes up (3 to 4), which is why the total numbers fall by 48, not 64. Convolutions and per-position layers scale with the number of positions (about 64 times cheaper); only the self-attention over all positions scales with its square (about 4,096 times). A real U-Net uses attention only at its coarser resolutions and is mostly convolution, so the realistic saving is large (tens of times) but nowhere near 2304. The notes themselves say "approximate"; treat 48 times fewer numbers as the solid fact.

6.3Text conditioning by cross-attention

The prompt is turned into vectors by a text encoder (CLIP in Stable Diffusion). Inside the U-Net, the denoising network uses cross-attention (Week 9): the queries $Q$ come from the U-Net's image features and the keys and values $K,V$ come from the text encoder's output. So at every attention layer each image position looks at the words of the prompt and takes what is relevant, steering generation at every layer. This is the same cross-attention as in the Week 10 decoder, with the "encoder" being the text. Combined with classifier-free guidance (Chapter 5, where the null condition is the empty prompt), this is the recipe of Stable Diffusion.

6.4Chapter summary

• LDM: train a VAE first and freeze it; train the diffusion model in the latent space; generate by denoising $z_T$ and decoding $z_0$.
• $512\times512\times3=786{,}432$ against $64\times64\times4=16{,}384$: 48 times fewer numbers, 64 times fewer positions; the notes' "2304 times" holds only for attention.
• Text conditioning: CLIP embedding feeds cross-attention ($Q$ from the U-Net, $K,V$ from the text) at every layer; with CFG this is Stable Diffusion.

What is likely to be asked from Week 11, with 30-second answers, deeper explanations, a lecture-or-extra tag on each question, and code you can run.

Week 11 questions are about how an image is destroyed, how the network learns to undo it, and the tricks (DDIM, guidance, latent space) that make it usable. Each answer starts with a 30-second version. Questions are tagged From the notes or Extra.

7.1Interview heat-map: what is most likely

7.2Foundations

the answer that minimises squared error: -0.001, an 'average' of the two correct answers
  squared error of the average         : 1.000
  squared error of committing randomly: 2.002   <- squared error PREFERS the average
distance from the average to the nearest real answer: 0.999 (it is not a valid answer at all)
so a decoder trained with MSE (a plain VAE) learns the mean of everything that could fit, which in an image is a blur;
a GAN or a diffusion model must commit to ONE sharp answer instead

after 500 steps: iterated mean 0.4226, std 0.9606   |   closed form mean 0.4245, std 0.9616   (theory: mean 0.4205, std 0.9599)
one jump to any t gives the same distribution as t small steps: training never has to iterate

alpha_bar (how much of the original signal's variance is left)
     t      linear    cosine
    50     0.9710    0.9920
   100     0.8970    0.9721
   250     0.5241    0.8470
   500     0.0786    0.4938
   750     0.0034    0.1443
   900     0.0003    0.0241
  1000     0.0000    0.0000
linear destroys the image early: by t = 500 only 0.079 of the signal is left, so the last half of the steps work on almost pure noise

Inception Score = exp( mean KL( P(y|x) || P(y) ) )   (1 is the worst, the number of classes is the best)
  sharp images, all 10 classes   :   9.58   <- realistic AND diverse
  sharp images, one class only   :   1.00   <- mode collapse is caught
  blurry images (P(y|x) uniform) :   1.00   <- unrealistic is caught
IS never looks at the real images, which is its limitation; FID (Week 6) compares generated features with real features

7.3Training and sampling

posterior mean (notes' formula)       : [ 0.3783  1.1056  1.1283 -0.5818 -1.3662]
mean written through the noise epsilon: [ 0.3783  1.1056  1.1283 -0.5818 -1.3662]
identical: True  -> predicting the NOISE is enough to take a denoising step
posterior variance beta_tilde = 0.008052  (beta_t = 0.008068)

cosine similarity of the embedding of step 10 with others (nearby steps should look alike):
   step  10 vs step  11: +0.966
   step  10 vs step  12: +0.884
   step  10 vs step  20: +0.658
   step  10 vs step  50: +0.481
   step  10 vs step 500: +0.216
   step  10 vs step 900: +0.155

DDPM                100            0.985          0.10 / 0.14
DDIM 50 steps        50            0.956          0.08 / 0.19
DDIM 20 steps        20            0.915          0.06 / 0.20
DDIM 10 steps        10            0.762          0.04 / 0.19
DDIM 5 steps          5            0.273          0.00 / 0.15
DDIM 2 steps          2            0.014          0.00 / 0.01

same starting noise x_T:
  DDPM run twice with different step noise: mean difference between the two results = 2.585 (stochastic)
  DDIM run twice:                          mean difference between the two results = 0.0 (deterministic)

DDPM                100            0.985          0.10 / 0.14
DDIM 50 steps        50            0.956          0.08 / 0.19
DDIM 20 steps        20            0.915          0.06 / 0.20
DDIM 10 steps        10            0.762          0.04 / 0.19
DDIM 5 steps          5            0.273          0.00 / 0.15
DDIM 2 steps          2            0.014          0.00 / 0.01

7.4Control and scale

real class-1 data: mean x +0.99, std x 1.00, fraction with x > 0: 0.842

guidance scale w   mean x   std x   fraction x > 0   std y
         0        -0.11    1.39        0.465      1.05
         1        +0.95    0.97        0.832      0.99
         2        +1.43    0.75        0.978      0.95
         4        +1.78    0.60        0.999      0.88
         8        +1.89    0.47        1.000      0.75
w = 0 is unconditional (both classes); w = 1 reproduces the real class-1 distribution; w > 1 pushes further and further into class 1 and the spread shrinks

real class-1 data: mean x +0.99, std x 1.00, fraction with x > 0: 0.842

guidance scale w   mean x   std x   fraction x > 0   std y
         0        -0.11    1.39        0.465      1.05
         1        +0.95    0.97        0.832      0.99

pixel space : 512 x 512 x 3 = 786,432 numbers
latent space: 64 x 64 x 4   = 16,384 numbers   -> 48x fewer
spatial positions: 262,144 against 4,096 = 64x fewer (the VAE shrinks height and width by 8)

what this does to the cost of the denoising network:
  operations that grow with the number of positions (convolutions, FFN): about 64x cheaper
  operations that grow with positions squared (self-attention over all positions): about 4096x cheaper
the notes' 2304x is the attention-only figure; a U-Net is mostly convolutions, so the realistic saving is a good deal smaller than 2304x
memory of one float32 image: 3.1 MB in pixel space, 65.5 KB as a latent

Week 11 on one page: chapter glance, every formula, the two samplers side by side, the pipeline of Stable Diffusion, a checklist and corrections to the source notes.

8.1Week 11 at a glance

8.2Every formula

8.3DDPM and DDIM side by side

8.4Key terms

8.5The pairs you must tell apart

8.6Last-minute checklist

8.7Corrections and additions to the source notes