Deep Learning,
Week 5

Every network in this course is trained the same way: define a loss that measures how wrong the model is, then change the weights to make it smaller. That second step is optimisation, and this whole week is about doing it well. Before the algorithms, two ideas about what we are really minimising.

1.1True risk and empirical risk

What we want is a model that does well on all the data the world could throw at it: the true risk $R(\theta)=\mathbb{E}_{(x,y)}[\ell(f_\theta(x),y)]$. We can never compute it, because we never see all the data. What we can compute is the average loss over the $n$ training examples we have: the empirical risk

Training minimises $\hat R$ and hopes that $R$ comes down with it. Sometimes it does not. Here are three polynomials fitted to 10 noisy points of a sine wave; the empirical risk is measured on those 10 points, and the true risk is estimated on 20,000 fresh points:

import numpy as np
rng = np.random.default_rng(0)
f_true = np.sin                                       # the (unknown) truth we want to learn
x_tr = rng.uniform(0, 2 * np.pi, 10); y_tr = f_true(x_tr) + rng.normal(0, 0.2, 10)   # 10 noisy training points
x_new = rng.uniform(0, 2 * np.pi, 20000); y_new = f_true(x_new) + rng.normal(0, 0.2, 20000)   # fresh data stands in for the whole distribution

for degree in (1, 3, 9):
    coef = np.polyfit(x_tr, y_tr, degree)
    emp = np.mean((np.polyval(coef, x_tr) - y_tr) ** 2)          # empirical risk: the loss we actually minimise
    true = np.mean((np.polyval(coef, x_new) - y_new) ** 2)       # estimate of the true risk: the loss we care about
    print(f"degree {degree}: empirical risk {emp:10.4f}   true risk (estimated) {true:12.4f}   gap {true - emp:12.4f}")

degree 1: empirical risk     0.2472   true risk (estimated)       0.4075   gap       0.1604
degree 3: empirical risk     0.0238   true risk (estimated)       0.0551   gap       0.0313
degree 9: empirical risk     0.0000   true risk (estimated)       0.3838   gap       0.3838

The degree-9 polynomial reaches an empirical risk of 0.0000, the best possible, and a true risk of 0.38, seven times worse than the modest degree-3 fit (0.055). Minimising the empirical risk to zero is therefore not the goal; it is a proxy, and the gap between the two is overfitting (Week 4). Optimisation tells us how to minimise; it does not tell us that we should minimise all the way.

1.2Batch, stochastic and mini-batch gradient descent

The gradient of $\hat R$ is an average over all $n$ examples. Computing it exactly means a pass over the whole dataset for one small step, which is far too slow for millions of images. Three ways to estimate it:

An epoch is one pass over the whole training set; an iteration is one weight update (one mini-batch). With $n=10{,}000$ and $B=100$ there are 100 iterations per epoch. In practice "SGD" almost always means the mini-batch version.

1.3How noisy is a mini-batch gradient? Measured

import numpy as np
rng = np.random.default_rng(0)
N = 20000
X = rng.normal(size=N); y = 3 * X + rng.normal(scale=1.0, size=N)
w = 0.0                                                        # gradient of the squared error at w = 0 is -2 x y
full = np.mean(-2 * X * (y - w * X))
print("full-batch gradient:", round(full, 3))
print("batch size   mean of mini-batch gradients   std of the mini-batch gradient   std x sqrt(B)")
for B in (1, 8, 64, 512, 4096):
    grads = []
    for _ in range(2000):
        idx = rng.integers(0, N, B)
        grads.append(np.mean(-2 * X[idx] * (y[idx] - w * X[idx])))
    grads = np.array(grads)
    print(f"{B:10d}   {grads.mean():28.3f}   {grads.std():30.3f}   {grads.std() * np.sqrt(B):12.2f}")
print("same mean (an unbiased estimate), but the noise falls like 1/sqrt(B): that is why a bigger batch tolerates a bigger learning rate")

full-batch gradient: -5.959
batch size   mean of mini-batch gradients   std of the mini-batch gradient   std x sqrt(B)
         1                         -5.848                            8.402           8.40
         8                         -5.898                            3.015           8.53
        64                         -5.947                            1.062           8.49
       512                         -5.962                            0.381           8.61
      4096                         -5.960                            0.140           8.93
same mean (an unbiased estimate), but the noise falls like 1/sqrt(B): that is why a bigger batch tolerates a bigger learning rate

Two facts are visible. The mean of the mini-batch gradients matches the full-batch one at every batch size (about $-5.96$): the estimate is unbiased. The spread falls like $1/\sqrt{B}$: the last column, std $\times\sqrt{B}$, stays near 8.5 instead of shrinking. So a batch 64 times larger has only 8 times less noise: bigger batches pay a lot of compute for a slowly shrinking benefit. A little noise is not even bad: it can shake the weights out of poor spots, which is one reason small-batch training often generalises well.

1.4Chapter summary

• We minimise the empirical risk $\hat R$ as a proxy for the true risk $R$; driving $\hat R$ to 0 can make $R$ worse (0.0000 vs 0.38 measured).
• Batch GD: exact, slow. SGD: noisy, cheap. Mini-batch: the practical compromise.
• Mini-batch gradient: unbiased, variance $\propto1/B$ (noise falls by $\sqrt{B}$).

Gradient descent walks downhill. Whether that walk ends somewhere good depends on the shape of the landscape, and the vocabulary for shapes is convex, local minimum and saddle point.

2.1Convex functions

A function is convex if the straight line (chord) between any two points of its graph never goes below the graph:

Test it by brute force: draw 100,000 random chords and count how often the chord dips below the curve.

import numpy as np
rng = np.random.default_rng(1)
def violations(f, trials=100000):
    x, y = rng.uniform(-3, 3, (2, trials)); lam = rng.uniform(0, 1, trials)
    return int(np.sum(f(lam * x + (1 - lam) * y) > lam * f(x) + (1 - lam) * f(y) + 1e-12))
print("chord above the curve fails for x^2   :", violations(lambda t: t ** 2), "of 100000 random chords (it is convex)")
print("chord above the curve fails for |x|   :", violations(np.abs), "(convex, not smooth)")
print("chord above the curve fails for sin(x):", violations(np.sin), "(NOT convex)")
print("chord above the curve fails for x^3   :", violations(lambda t: t ** 3), "(NOT convex on all of R)")

chord above the curve fails for x^2   : 0 of 100000 random chords (it is convex)
chord above the curve fails for |x|   : 0 (convex, not smooth)
chord above the curve fails for sin(x): 50034 (NOT convex)
chord above the curve fails for x^3   : 50104 (NOT convex on all of R)

$x^2$ and $|x|$ never break the rule (note $|x|$ has a corner: convex does not need smooth). $\sin x$ breaks it about half the time, and $x^3$ too (it bends the wrong way for $x<0$). For a twice-differentiable function there is an easier test: convex $\iff$ the Hessian is positive semi-definite everywhere (all eigenvalues $\ge0$, the curvature never points down).

import torch
H = torch.autograd.functional.hessian
cases = {
    "x^2 + y^2          ": lambda p: p[0] ** 2 + p[1] ** 2,
    "x^2 + 5 y^2        ": lambda p: p[0] ** 2 + 5 * p[1] ** 2,
    "x^2 - y^2 (saddle) ": lambda p: p[0] ** 2 - p[1] ** 2,
    "x^4 - 3 x^2 + y^2  ": lambda p: p[0] ** 4 - 3 * p[0] ** 2 + p[1] ** 2,
}
for name, f in cases.items():
    for pt in ([0.0, 0.0], [1.5, 0.0]):
        eig = torch.linalg.eigvalsh(H(f, torch.tensor(pt)))
        verdict = "locally convex here" if bool((eig >= -1e-6).all()) else "NOT convex here"
        print(f"f = {name} at {pt}: Hessian eigenvalues {[round(v, 2) for v in eig.tolist()]} -> {verdict}")
print("a function is convex iff its Hessian is positive semi-definite EVERYWHERE; x^4-3x^2+y^2 fails at the origin")

f = x^2 + y^2           at [0.0, 0.0]: Hessian eigenvalues [2.0, 2.0] -> locally convex here
f = x^2 + y^2           at [1.5, 0.0]: Hessian eigenvalues [2.0, 2.0] -> locally convex here
f = x^2 + 5 y^2         at [0.0, 0.0]: Hessian eigenvalues [2.0, 10.0] -> locally convex here
f = x^2 + 5 y^2         at [1.5, 0.0]: Hessian eigenvalues [2.0, 10.0] -> locally convex here
f = x^2 - y^2 (saddle)  at [0.0, 0.0]: Hessian eigenvalues [-2.0, 2.0] -> NOT convex here
f = x^2 - y^2 (saddle)  at [1.5, 0.0]: Hessian eigenvalues [-2.0, 2.0] -> NOT convex here
f = x^4 - 3 x^2 + y^2   at [0.0, 0.0]: Hessian eigenvalues [-6.0, 2.0] -> NOT convex here
f = x^4 - 3 x^2 + y^2   at [1.5, 0.0]: Hessian eigenvalues [2.0, 21.0] -> locally convex here
a function is convex iff its Hessian is positive semi-definite EVERYWHERE; x^4-3x^2+y^2 fails at the origin

The last function $x^4-3x^2+y^2$ is convex near $x=1.5$ (eigenvalues 2 and 21) but not at the origin (an eigenvalue of $-6$), so it is not convex: a function must pass the test everywhere. The saddle $x^2-y^2$ fails everywhere.

Jensen's inequality

The chord definition, extended from two points to a whole distribution, gives Jensen's inequality: for convex $f$, $f(\mathbb{E}[X])\le\mathbb{E}[f(X)]$, and the inequality flips for concave $f$. It appears in later weeks (for example in the bound that GANs and diffusion models rest on), so see it once with numbers:

import numpy as np
rng = np.random.default_rng(0)
X = rng.normal(2.0, 1.5, 100000)                      # a random variable with mean 2, variance 2.25
f = lambda x: x ** 2                                  # convex
print("f(E[X]) =", round(f(X.mean()), 4), "   E[f(X)] =", round(f(X).mean(), 4))
print("f(E[X]) <= E[f(X)]:", f(X.mean()) <= f(X).mean(), "  (the gap is exactly Var(X) =", round(X.var(), 3), ")")
g = np.log                                            # concave: the inequality flips
Y = rng.uniform(1, 5, 100000)
print("log: f(E[Y]) =", round(g(Y.mean()), 4), ">= E[f(Y)] =", round(g(Y).mean(), 4))

f(E[X]) = 3.9946    E[f(X)] = 6.2451
f(E[X]) <= E[f(X)]: True   (the gap is exactly Var(X) = 2.251 )
log: f(E[Y]) = 1.0977 >= E[f(Y)] = 1.0112

For $f(x)=x^2$ the gap $\mathbb{E}[X^2]-(\mathbb{E}X)^2$ is exactly the variance (2.251 here), which is also why variance is never negative.

2.2Local minima and the non-convex world

# f(x) = x^4 - 3x^2 + x has TWO local minima; gradient descent ends in the one nearest its start
f = lambda x: x ** 4 - 3 * x ** 2 + x
df = lambda x: 4 * x ** 3 - 6 * x + 1
for start in (-2.0, -0.5, 0.5, 2.0):
    x = start
    for _ in range(2000):
        x -= 0.01 * df(x)
    print(f"start {start:+.1f}  ->  ends at x = {x:+.4f}, f = {f(x):+.4f}")
print("two different answers from the same function: the left one (f = -3.514) is the global minimum, the right one (f = -1.070) is only a local minimum")

start -2.0  ->  ends at x = -1.3008, f = -3.5139
start -0.5  ->  ends at x = -1.3008, f = -3.5139
start +0.5  ->  ends at x = +1.1309, f = -1.0702
start +2.0  ->  ends at x = +1.1309, f = -1.0702
two different answers from the same function: the left one (f = -3.514) is the global minimum, the right one (f = -1.070) is only a local minimum

The same algorithm, the same learning rate, two different answers, decided only by the starting point. In real networks there are astronomically many such valleys, and the surprising empirical finding of deep learning is that most reachable minima are about equally good, so getting stuck in a bad one is rarer than the picture suggests. The troublemaker in high dimensions is a different shape.

2.3Saddle points

A saddle point has a zero gradient but is neither a minimum nor a maximum: it curves up in some directions and down in others (like a horse saddle or a mountain pass). The Hessian there has both positive and negative eigenvalues. Check it, then run descent from two starts:

import torch
f = lambda p: p[0] ** 2 - p[1] ** 2                      # a minimum along x, a maximum along y: a saddle
p0 = torch.zeros(2, requires_grad=True)
f(p0).backward()
print("gradient at the origin:", p0.grad.tolist(), " <- zero, yet the origin is NOT a minimum")
H = torch.autograd.functional.hessian(f, torch.zeros(2))
print("Hessian eigenvalues   :", torch.linalg.eigvalsh(H).tolist(), " <- one negative: some direction goes down")

def descend(start, steps=60, lr=0.1):
    p = torch.tensor(start)
    for _ in range(steps):
        g = torch.stack([2 * p[0], -2 * p[1]])
        p = p - lr * g
    return p.tolist()
print("start (1, 0.000) exactly on the ridge ->", [round(v, 6) for v in descend([1.0, 0.0])], " (walks into the saddle and stops)")
print("start (1, 0.001) a hair off the ridge ->", [round(v, 2) for v in descend([1.0, 0.001])], " (slides away, but slowly at first)")

gradient at the origin: [0.0, 0.0]  <- zero, yet the origin is NOT a minimum
Hessian eigenvalues   : [-2.0, 2.0]  <- one negative: some direction goes down
start (1, 0.000) exactly on the ridge -> [2e-06, 0.0]  (walks into the saddle and stops)
start (1, 0.001) a hair off the ridge -> [0.0, 56.35]  (slides away, but slowly at first)

Two lessons. Zero gradient does not mean minimum: always say "critical point", and use the Hessian (all eigenvalues positive: minimum; mixed: saddle; all negative: maximum) to classify it. And descent does escape a saddle, but only if it is nudged off the ridge: from exactly $y=0$ it stops dead at the saddle, from $y=0.001$ the escape direction grows by 1.2 each step ($0.001\times1.2^{60}\approx56$), slow at first and then fast. A long, flat plateau near a saddle is where training seems to stall for ages.

2.4Why saddles, not minima, dominate in high dimensions

At a critical point in $d$ dimensions there are $d$ curvature directions. For it to be a true minimum all $d$ must curve upward. If each is roughly a coin flip, that is a $2^{-d}$ event. A quick simulation with random symmetric matrices as stand-in Hessians:

import numpy as np
rng = np.random.default_rng(0)
# Model the Hessian at a critical point as a random symmetric matrix.
# It is a local minimum only if ALL eigenvalues are positive.
print(" dim   P(all eigenvalues > 0)   (rest are saddles or maxima)")
for d in (1, 2, 3, 5, 8, 12, 20):
    ok = 0
    trials = 4000
    for _ in range(trials):
        A = rng.normal(size=(d, d)); H = (A + A.T) / 2
        ok += bool(np.all(np.linalg.eigvalsh(H) > 0))
    print(f"{d:4d}   {ok / trials:8.4f}")

 dim   P(all eigenvalues > 0)   (rest are saddles or maxima)
   1     0.4868
   2     0.1547
   3     0.0240
   5     0.0000
   8     0.0000
  12     0.0000
  20     0.0000

The chance that all eigenvalues are positive collapses: 49% in one dimension, 2.4% in three, and below 1 in 4000 from five dimensions up. A network has millions of dimensions, so a random critical point is almost surely a saddle, not a minimum. Honest caveat: real Hessians are not random matrices, so this is an intuition for why saddles are common, not a measurement of any network. It is the standard argument for why saddles, not local minima, are the main obstacle in deep learning, and why algorithms that keep moving (momentum, adaptive methods) help.

2.5Chapter summary

• Convex: chords stay above the graph; Hessian PSD everywhere. Then every local minimum is global.
• Neural-network losses are non-convex: the end point depends on the start (two minima, -3.514 vs -1.070).
• Zero gradient = critical point; classify with the Hessian. Saddles (mixed signs) dominate in high dimensions.
• Jensen: convex $f(\mathbb{E}X)\le\mathbb{E}f(X)$; the gap for $x^2$ is the variance.

The basic update, repeated: move a little against the gradient, $\theta\leftarrow\theta-\eta\,\nabla L(\theta)$. The only knob is the learning rate $\eta$. Too small is slow; too large is not just fast and sloppy, it is unstable. Here is exactly where the line is.

3.1The learning rate and its stability limit

Take the simplest bowl, $L=\tfrac12\lambda w^2$, whose gradient is $\lambda w$. One step gives $w\leftarrow(1-\eta\lambda)\,w$. The weight shrinks to zero only if $|1-\eta\lambda|<1$, that is $\eta<2/\lambda$. A steeper bowl (bigger curvature $\lambda$) tolerates a smaller step. If there are several directions with curvatures $\lambda_{\min}\ldots\lambda_{\max}$, the steepest one sets the limit, $\eta<2/\lambda_{\max}$, while the flattest one sets the speed.

Test it on an elongated bowl with curvatures 1 and 50 (rotated 45 degrees, so no direction is special). The predicted limit is $2/50=0.04$:

import numpy as np
# an elongated bowl: curvature 1 in one direction and 50 in the other, rotated 45 degrees
th = np.pi / 4; Q = np.array([[np.cos(th), -np.sin(th)], [np.sin(th), np.cos(th)]])
A = Q @ np.diag([1.0, 50.0]) @ Q.T
f = lambda p: 0.5 * p @ A @ p
g = lambda p: A @ p

def steps_to_converge(lr, max_steps=20000):
    p = np.array([1.0, 0.3])
    for t in range(1, max_steps + 1):
        p = p - lr * g(p)
        if not np.isfinite(p).all() or np.linalg.norm(p) > 1e6:
            return None
        if np.linalg.norm(g(p)) < 1e-3:
            return t
    return None

print("curvatures 1 and 50, so a stable step needs lr < 2/50 = 0.04")
for lr in (0.001, 0.01, 0.02, 0.039, 0.0399, 0.041):
    n = steps_to_converge(lr)
    print(f"lr = {lr:<7}: " + (f"converged in {n} steps" if n else "did not converge (diverged)"))

curvatures 1 and 50, so a stable step needs lr < 2/50 = 0.04
lr = 0.001  : converged in 6821 steps
lr = 0.01   : converged in 679 steps
lr = 0.02   : converged in 338 steps
lr = 0.039  : converged in 199 steps
lr = 0.0399 : converged in 2019 steps
lr = 0.041  : did not converge (diverged)

Exactly as predicted: below 0.04 it converges, at 0.041 it diverges. Two more details. Going from $\eta=0.001$ to 0.01 to 0.02 to 0.039 cuts the steps from 6821 to 679 to 338 to 199, because the slow, flat direction needs a big step. But at 0.0399, just under the limit, it needs 2019 steps, ten times more: the steep direction is multiplied by $1-0.0399\times50=-0.995$ each step and so flips sign and almost never dies away. The best step is not "as large as stable" but $\eta^*=2/(\lambda_{\min}+\lambda_{\max})=2/51\approx0.039$.

3.2Conditioning: why elongated bowls are hard

The ratio $\kappa=\lambda_{\max}/\lambda_{\min}$ is the condition number. At the best step, each iteration shrinks the error by the factor $(\kappa-1)/(\kappa+1)$. For $\kappa=50$ that is $49/51=0.961$, and $0.961^{199}\approx3.5\times10^{-4}$: the 199 steps above. For $\kappa=1$ (a round bowl) one step lands on the minimum; for $\kappa=10^4$ you would need about 100 times more steps. Plain gradient descent is slow whenever the landscape is much steeper in one direction than in another, which is exactly the shape of real loss surfaces, and all of this week's improvements attack that.

3.3The second-order answer: Newton's method

If the problem is that the gradient ignores curvature, use the curvature: $\theta\leftarrow\theta-H^{-1}\nabla L$, where $H$ is the Hessian. On a quadratic bowl this lands on the exact minimum in one step, whatever $\kappa$. So why is it not used to train networks?

import numpy as np
th = np.pi / 4; Q = np.array([[np.cos(th), -np.sin(th)], [np.sin(th), np.cos(th)]])
A = Q @ np.diag([1.0, 50.0]) @ Q.T                    # f(p) = 0.5 p^T A p, Hessian = A
p = np.array([1.0, 0.3])
p_new = p - np.linalg.solve(A, A @ p)                 # Newton: p - H^{-1} grad
print("one Newton step on an ill-conditioned quadratic:", p, "->", p_new.round(12), " (exact minimum in ONE step)")

# on the saddle f = x^2 - y^2 Newton is attracted to the saddle itself
H = np.diag([2.0, -2.0]); q = np.array([1.0, 1.0]); grad = np.array([2.0 * q[0], -2.0 * q[1]])
print("Newton on the saddle x^2 - y^2 from (1, 1):", (q - np.linalg.solve(H, grad)).tolist(), " <- lands ON the saddle, not a minimum")

for d in (1_000, 1_000_000, 100_000_000):
    gb = d * d * 4 / 1e9
    print(f"d = {d:>11,} parameters: Hessian has {d * d:.0e} entries = {gb:,.3f} GB in float32" if gb < 1 else f"d = {d:>11,} parameters: Hessian has {d * d:.0e} entries = {gb:,.0f} GB in float32")

one Newton step on an ill-conditioned quadratic: [1.  0.3] -> [-0. -0.]  (exact minimum in ONE step)
Newton on the saddle x^2 - y^2 from (1, 1): [0.0, 0.0]  <- lands ON the saddle, not a minimum
d =       1,000 parameters: Hessian has 1e+06 entries = 0.004 GB in float32
d =   1,000,000 parameters: Hessian has 1e+12 entries = 4,000 GB in float32
d = 100,000,000 parameters: Hessian has 1e+16 entries = 40,000,000 GB in float32

• Cost. The Hessian has $d^2$ entries: 4,000 GB for a million parameters, tens of millions of GB for a hundred million. Inverting it costs about $d^3$.
• Saddles attract it. Newton looks for a point where the gradient is zero, so on $x^2-y^2$ it jumps straight to the saddle (the output above) instead of leaving it.
• Noise. With a noisy mini-batch gradient and a noisy Hessian the "exact" step is not exact.

So deep learning keeps the first-order machinery and borrows curvature information cheaply: momentum smooths the path, and adaptive methods rescale each weight by a running estimate of its gradient size. That is the next two chapters.

3.4Chapter summary

• Update $\theta\leftarrow\theta-\eta\nabla L$; stable only if $\eta<2/\lambda_{\max}$ (measured: diverges at 0.041, limit 0.04).
• Best step $2/(\lambda_{\min}+\lambda_{\max})$; error shrinks by $(\kappa-1)/(\kappa+1)$ per step, so large $\kappa$ means slow.
• Newton fixes conditioning in one step but costs $d^2$ memory, $d^3$ time and is drawn to saddles.

On an elongated bowl, plain gradient descent bounces across the narrow direction and crawls along the long one. Look at the paths.

4.1Momentum: remember where you were going

Picture a heavy ball rolling down the bowl instead of a hiker who stops and re-plans at every step. The ball keeps a velocity $v$, a running blend of past gradients:

With $\beta=0.9$ the velocity remembers roughly the last $1/(1-\beta)=10$ gradients. Two effects follow. In the narrow direction the gradient alternates in sign (left, right, left), so the contributions cancel in the average and the bouncing dies down. In the long direction the gradient keeps the same sign, so the contributions add up: the ball accelerates, and in steady motion its step is about $\eta/(1-\beta)$ $=10\eta$, as if the learning rate had been multiplied by ten, but only where the gradient is consistent.

4.2Nesterov: look before you leap

Momentum computes the gradient where the ball is. Nesterov momentum computes it where the velocity is about to take the ball: $\nabla L(\theta+\beta v)$. If the ball is about to overshoot the valley floor, the look-ahead gradient already points back, so it brakes earlier.

4.3Measured: how much faster?

The same elongated bowl as Chapter 3, the same starting point, the same learning rate of 0.01 for all three. Steps until the gradient norm drops below 0.001:

import numpy as np
th = np.pi / 4; Q = np.array([[np.cos(th), -np.sin(th)], [np.sin(th), np.cos(th)]])
A = Q @ np.diag([1.0, 50.0]) @ Q.T
g = lambda p: A @ p
START = np.array([1.0, 0.3])

def run(step, init, max_steps=20000):
    p, s = START.copy(), init()
    for t in range(1, max_steps + 1):
        p = step(p, s, t)
        if np.linalg.norm(g(p)) < 1e-3:
            return t
    return None

def gd(lr):       return (lambda p, s, t: p - lr * g(p)), (lambda: None)
def momentum(lr, b):
    def step(p, s, t): s["v"] = b * s["v"] - lr * g(p); return p + s["v"]
    return step, lambda: {"v": np.zeros(2)}
def nesterov(lr, b):
    def step(p, s, t): s["v"] = b * s["v"] - lr * g(p + b * s["v"]); return p + s["v"]
    return step, lambda: {"v": np.zeros(2)}
def adagrad(lr):
    def step(p, s, t): gg = g(p); s["s"] += gg ** 2; return p - lr * gg / (np.sqrt(s["s"]) + 1e-8)
    return step, lambda: {"s": np.zeros(2)}
def rmsprop(lr, b=0.9):
    def step(p, s, t): gg = g(p); s["s"] = b * s["s"] + (1 - b) * gg ** 2; return p - lr * gg / (np.sqrt(s["s"]) + 1e-8)
    return step, lambda: {"s": np.zeros(2)}
def adam(lr, b1=0.9, b2=0.999):
    def step(p, s, t):
        gg = g(p); s["m"] = b1 * s["m"] + (1 - b1) * gg; s["v"] = b2 * s["v"] + (1 - b2) * gg ** 2
        return p - lr * (s["m"] / (1 - b1 ** t)) / (np.sqrt(s["v"] / (1 - b2 ** t)) + 1e-8)
    return step, lambda: {"m": np.zeros(2), "v": np.zeros(2)}

rows = [("plain GD, best stable lr 0.039", gd(0.039)), ("plain GD, lr 0.01", gd(0.01)),
        ("momentum 0.9, lr 0.01", momentum(0.01, 0.9)), ("Nesterov 0.9, lr 0.01", nesterov(0.01, 0.9)),
        ("Adagrad lr 0.5", adagrad(0.5)), ("RMSProp lr 0.05", rmsprop(0.05)), ("Adam lr 0.1", adam(0.1)), ("Adam lr 0.05", adam(0.05))]
print("steps until the gradient norm is below 1e-3 (condition number 50):")
for name, (step, init) in rows:
    n = run(step, init)
    print(f"  {name:32s}", n if n else "not reached (keeps jittering at the step size)")

steps until the gradient norm is below 1e-3 (condition number 50):
  plain GD, best stable lr 0.039   199
  plain GD, lr 0.01                679
  momentum 0.9, lr 0.01            95
  Nesterov 0.9, lr 0.01            62
  Adagrad lr 0.5                   263
  RMSProp lr 0.05                  not reached (keeps jittering at the step size)
  Adam lr 0.1                      147
  Adam lr 0.05                     303

At the same $\eta=0.01$: plain GD needs 679 steps, momentum 95 (7 times fewer), Nesterov 62 (11 times fewer). Momentum even beats plain GD at its very best step (199). Momentum also widens the stable range of learning rates, from $\eta<2/\lambda_{\max}$ to $\eta<2(1+\beta)/\lambda_{\max}$:

import numpy as np
th = np.pi / 4; Q = np.array([[np.cos(th), -np.sin(th)], [np.sin(th), np.cos(th)]])
A = Q @ np.diag([1.0, 50.0]) @ Q.T                       # curvatures 1 and 50, as before
beta = 0.9
print("plain GD is stable for lr < 2/50 = 0.04; heavy-ball momentum for lr < 2(1+beta)/50 =", round(2 * (1 + beta) / 50, 3))
for lr in (0.05, 0.07, 0.075, 0.078):
    p = np.array([1.0, 0.3]); v = np.zeros(2); result = "not converged in 20000 steps"
    for t in range(1, 20001):
        v = beta * v - lr * A @ p; p = p + v
        if not np.isfinite(p).all() or np.linalg.norm(p) > 1e6:
            result = "diverged"; break
        if np.linalg.norm(A @ p) < 1e-3:
            result = f"converged in {t} steps"; break
    print(f"momentum 0.9, lr = {lr}: {result}")

plain GD is stable for lr < 2/50 = 0.04; heavy-ball momentum for lr < 2(1+beta)/50 = 0.076
momentum 0.9, lr = 0.05: converged in 158 steps
momentum 0.9, lr = 0.07: converged in 149 steps
momentum 0.9, lr = 0.075: converged in 182 steps
momentum 0.9, lr = 0.078: diverged

With $\beta=0.9$ the limit moves from 0.04 to 0.076, and the run confirms it (0.075 converges, 0.078 diverges). On non-convex problems the benefit is also that the ball carries enough speed to roll through shallow dips and flat plateaus near saddles, and a large $\beta$ trades this for overshoot: the ball swings past the bottom and oscillates before it settles.

4.4Chapter summary

• Momentum: $v\leftarrow\beta v-\eta g,\ \theta\leftarrow\theta+v$. Cancels zigzag across the steep direction, accumulates along the flat one.
• Nesterov takes the gradient at the look-ahead point $\theta+\beta v$.
• Measured at the same lr: 679 (GD), 95 (momentum), 62 (Nesterov) steps.

Momentum fixes the direction problem. A second problem is that one learning rate is shared by every weight, though weights differ wildly: a weight on a rare feature gets a gradient only now and then, a weight on a common feature gets one every step. Adaptive methods give each weight its own step size, scaled by how large its gradients have been.

5.1Adagrad: divide by the history

$s_t$ is the running sum of squared gradients for that weight. A weight that has seen big gradients gets a small effective step; one that has seen tiny gradients keeps a large step. It suits sparse data (a rare word's embedding keeps a large step for the day it finally appears) and needs almost no tuning of $\eta$. Its flaw is that $s_t$ only ever grows:

import numpy as np
lr = 0.1
print("a constant gradient g = 1 at every step")
print("   step t    accumulated sum    effective lr = lr / sqrt(sum)")
for t in (1, 10, 100, 1000, 10000):
    s = t * 1.0 ** 2                                     # sum of squared gradients grows linearly
    print(f"{t:9d}    {s:13.0f}       {lr / np.sqrt(s):.6f}")
print("the learning rate shrinks like 1/sqrt(t) and never recovers: Adagrad can stall before it finishes")

beta = 0.9                                               # RMSProp: an exponential moving average instead of a running sum
s = 0.0
for t in range(1, 10001):
    s = beta * s + (1 - beta) * 1.0
print("RMSProp after 10000 steps of the same gradient: denominator", round(float(np.sqrt(s)), 4), "-> effective lr", round(lr / float(np.sqrt(s)), 4), "(stays near lr)")

a constant gradient g = 1 at every step
   step t    accumulated sum    effective lr = lr / sqrt(sum)
        1                1       0.100000
       10               10       0.031623
      100              100       0.010000
     1000             1000       0.003162
    10000            10000       0.001000
the learning rate shrinks like 1/sqrt(t) and never recovers: Adagrad can stall before it finishes
RMSProp after 10000 steps of the same gradient: denominator 1.0 -> effective lr 0.1 (stays near lr)

With a constant gradient the effective rate falls like $1/\sqrt{t}$: from 0.1 to 0.001 after 10,000 steps, and it never recovers. On a long deep-learning run Adagrad can slow to a crawl before it has finished learning.

5.2RMSProp and AdaDelta: forget the distant past

The fix is a one-word change: replace the sum with an exponential moving average, so old gradients fade out. In the output above, after 10,000 steps of the same gradient RMSProp's denominator is 1.0 and its effective rate is still 0.1, where Adagrad's had fallen to 0.001. AdaDelta uses the same moving average and additionally replaces $\eta$ by a running average of the size of past updates, so it needs no learning rate at all (it is rarely the first choice today).

5.3Adam: momentum plus RMSProp

$m$ is momentum (direction, the average of $g$), $v$ is RMSProp's scale (the average of $g^2$). Defaults: $\beta_1=0.9,\ \beta_2=0.999,\ \epsilon=10^{-8}$, and a learning rate around $10^{-3}$. The bias correction divides by $1-\beta^t$ because both averages start at zero and are therefore biased low for the first steps. Run three steps by hand:

import numpy as np
lr, b1, b2, eps = 0.01, 0.9, 0.999, 1e-8
m = v = 0.0
for t, g in enumerate([0.5, 0.5, 0.5], start=1):
    m = b1 * m + (1 - b1) * g                  # first moment: a moving average of the gradient
    v = b2 * v + (1 - b2) * g ** 2             # second moment: a moving average of the squared gradient
    m_hat = m / (1 - b1 ** t)                  # bias correction
    v_hat = v / (1 - b2 ** t)
    update = lr * m_hat / (np.sqrt(v_hat) + eps)
    print(f"t={t}: m={m:.5f}  v={v:.8f}  m_hat={m_hat:.3f}  v_hat={v_hat:.4f}  update={update:.6f}")
print()
print("without bias correction at t=1 the step would be", round(lr * (0.05) / (np.sqrt(0.00025) + eps), 5), "(3.16x too LARGE: v starts at 0 and is badly underestimated, more than m is)")
print("with correction, the very first step is lr * sign(g) =", lr, ": Adam's step size is about lr, whatever the gradient scale")
for scale in (1e-4, 1.0, 1e4):
    g = 0.5 * scale
    mh, vh = g, g ** 2
    print(f"gradient {g:10.4f}: first Adam step {lr * mh / (np.sqrt(vh) + eps):.6f}")

t=1: m=0.05000  v=0.00025000  m_hat=0.500  v_hat=0.2500  update=0.010000
t=2: m=0.09500  v=0.00049975  m_hat=0.500  v_hat=0.2500  update=0.010000
t=3: m=0.13550  v=0.00074925  m_hat=0.500  v_hat=0.2500  update=0.010000

without bias correction at t=1 the step would be 0.03162 (3.16x too LARGE: v starts at 0 and is badly underestimated, more than m is)
with correction, the very first step is lr * sign(g) = 0.01 : Adam's step size is about lr, whatever the gradient scale
gradient     0.0001: first Adam step 0.009998
gradient     0.5000: first Adam step 0.010000
gradient  5000.0000: first Adam step 0.010000

Look at what the correction does. Without it the very first step would be 0.0316, 3.16 times too large, because $v$ (weight 0.001 on $g^2$) is underestimated far more than $m$ (weight 0.1 on $g$). With it $\hat m=0.5$ and $\hat v=0.25$ are exact and the step is exactly $\eta$. And the last three lines show Adam's signature property: for gradients of $10^{-4}$, 0.5 and 5000 the first step is the same, about $\eta$. Adam's step size is set by the learning rate, not by the size of the gradient, which is why one value like $10^{-3}$ works across so many problems.

5.4AdamW: decoupled weight decay

Weight decay (Week 4) shrinks weights towards zero. In plain SGD, adding an L2 penalty $\tfrac{\lambda}{2}\|w\|^2$ to the loss and shrinking the weights directly are the same thing. In Adam they are not: the L2 gradient $\lambda w$ is added to $g$, and then divided by $\sqrt{\hat v}$ like everything else, so the strength of the decay now depends on the gradient history. AdamW removes the decay from the gradient and applies it separately: $\theta\leftarrow\theta-\eta\big(\hat m/(\sqrt{\hat v}+\epsilon)+\lambda\theta\big)$. Isolate the effect by giving the loss no gradient at all:

import torch
# Isolate the weight decay: the loss gradient is exactly zero, so the only force on the weight is the penalty.
# isolate the decay: no loss gradient, only decay
def decay_only(opt_cls):
    w = torch.nn.Parameter(torch.tensor([1.0, 1.0]))
    opt = opt_cls([w], lr=0.1, weight_decay=0.1)
    for _ in range(100):
        opt.zero_grad(); (0 * w).sum().backward(); opt.step()
    return [round(v, 3) for v in w.tolist()]
print("weights start at [1.0, 1.0], no loss gradient, weight_decay = 0.1, lr = 0.1, 100 steps")
print("decay only, Adam :", decay_only(torch.optim.Adam), " <- L2 sits inside the gradient, so Adam rescales it like any gradient: the weight is driven to ~0")
print("decay only, AdamW:", decay_only(torch.optim.AdamW), " <- plain shrinkage factor (1 - lr*wd)^100 =", round((1 - 0.1 * 0.1) ** 100, 3))

weights start at [1.0, 1.0], no loss gradient, weight_decay = 0.1, lr = 0.1, 100 steps
decay only, Adam : [0.003, 0.003]  <- L2 sits inside the gradient, so Adam rescales it like any gradient: the weight is driven to ~0
decay only, AdamW: [0.366, 0.366]  <- plain shrinkage factor (1 - lr*wd)^100 = 0.366

Same learning rate, same weight_decay=0.1, same 100 steps: Adam with L2 drives the weight to 0.003 (Adam rescaled the tiny penalty gradient up to a full step, every step), AdamW shrinks it by exactly the predictable factor $(1-\eta\lambda)^{100}=0.366$. AdamW's decay does what the hyper-parameter says, which is why it is the default for Transformers (Weeks 9-10).

5.5Chapter summary

• Adagrad: divide by $\sqrt{\text{sum of }g^2}$, the rate only shrinks. RMSProp: moving average, so it forgets. AdaDelta: no learning rate.
• Adam = momentum $m$ + RMSProp $v$ + bias correction; defaults 0.9, 0.999, $10^{-8}$; step $\approx\eta$ whatever the gradient size.
• AdamW decouples weight decay (0.366 vs 0.003 measured); use it rather than Adam with L2.

Two practical questions remain. How should the learning rate change during training? And which optimiser should I actually pick?

6.1Learning-rate schedules

A good learning rate early (large, to make progress and cross flat regions) is a bad one late (small is needed to settle into the valley instead of bouncing around it). A schedule lowers it over time:

• Step decay: multiply by a factor (here 0.1) every fixed number of epochs. Simple, a classic for image models.
• Exponential decay: $\eta_t=\eta_0e^{-kt}$, a smooth version of the same idea.
• Cosine annealing: $\eta_t=\tfrac12\eta_0\big(1+\cos(\pi t/T)\big)$, a gentle S-shaped fall to zero. Very popular today.
• Warm-up: start tiny and ramp up for the first few hundred steps, because the very first Adam steps use badly estimated statistics (Transformers almost always use it).

import math, torch
lr0, T = 0.1, 100
step_decay = lambda t: lr0 * 0.1 ** (t // 30)                          # x0.1 every 30 epochs
exp_decay = lambda t: lr0 * math.exp(-0.03 * t)
cosine = lambda t: 0.5 * lr0 * (1 + math.cos(math.pi * t / T))
print("epoch   step    exponential   cosine")
for t in (0, 15, 30, 60, 90, 100):
    print(f"{t:5d}   {step_decay(t):.4f}   {exp_decay(t):.4f}        {cosine(t):.4f}")

# the PyTorch schedulers produce the same cosine numbers
w = torch.nn.Parameter(torch.zeros(1))
opt = torch.optim.SGD([w], lr=lr0)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=T)
got = []
for t in range(T + 1):
    got.append(opt.param_groups[0]["lr"]); opt.step(); sched.step()
print("scheduler matches the formula:", all(abs(got[t] - cosine(t)) < 1e-9 for t in (0, 15, 30, 60, 90, 100)))

epoch   step    exponential   cosine
    0   0.1000   0.1000        0.1000
   15   0.1000   0.0638        0.0946
   30   0.0100   0.0407        0.0794
   60   0.0010   0.0165        0.0345
   90   0.0001   0.0067        0.0024
  100   0.0001   0.0050        0.0000
scheduler matches the formula: True

In PyTorch the same curves come from torch.optim.lr_scheduler (StepLR, ExponentialLR, CosineAnnealingLR); the last line confirms the library reproduces the formula. Call scheduler.step() once per epoch (or per iteration, depending on how you set it up), after optimizer.step().

6.2All the optimisers on one problem

On the elongated bowl (Chapter 3): momentum and Nesterov win clearly; Adam at lr 0.1 beats plain GD at its best (147 against 199) but at lr 0.05 it needs 303, so it is not tuning-free; and RMSProp at lr 0.05 never reaches the target because its step is about $\eta$ per weight regardless of the gradient, so it keeps jittering around the minimum at that size. That last point is a real property: sign-like methods need a decaying learning rate to settle, which is why Adam is usually paired with a schedule.

And on a real small network (two interleaved half-moons, a 2-32-32-1 ReLU net, full batch, the same initial weights): epochs until the training loss falls below 0.15.

import torch, torch.nn as nn, math
def make_moons(n, noise, seed):
    g = torch.Generator().manual_seed(seed)
    t = torch.rand(n, generator=g) * math.pi
    X = torch.cat([torch.stack([torch.cos(t), torch.sin(t)], 1), torch.stack([1 - torch.cos(t), 0.5 - torch.sin(t)], 1)]) + noise * torch.randn(2 * n, 2, generator=g)
    y = torch.cat([torch.zeros(n), torch.ones(n)]); p = torch.randperm(2 * n, generator=g)
    return X[p], y[p]
X, y = make_moons(400, 0.15, 0)

def epochs_to(make_opt, target=0.15, max_epochs=600):
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(2, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 1))
    opt = make_opt(model.parameters())
    for epoch in range(1, max_epochs + 1):
        opt.zero_grad()
        loss = nn.functional.binary_cross_entropy_with_logits(model(X).squeeze(1), y)
        loss.backward(); opt.step()
        if loss.item() < target:
            return epoch
    return None

opts = {
    "SGD lr 0.1":                 lambda p: torch.optim.SGD(p, lr=0.1),
    "SGD lr 0.1, momentum 0.9":   lambda p: torch.optim.SGD(p, lr=0.1, momentum=0.9),
    "SGD lr 0.1, Nesterov 0.9":   lambda p: torch.optim.SGD(p, lr=0.1, momentum=0.9, nesterov=True),
    "Adagrad lr 0.1":             lambda p: torch.optim.Adagrad(p, lr=0.1),
    "RMSprop lr 0.01":            lambda p: torch.optim.RMSprop(p, lr=0.01),
    "Adam lr 0.01":               lambda p: torch.optim.Adam(p, lr=0.01),
    "AdamW lr 0.01":              lambda p: torch.optim.AdamW(p, lr=0.01),
}
print("full-batch epochs until training loss < 0.15 (same network, same initial weights):")
for name, make in opts.items():
    n = epochs_to(make)
    print(f"  {name:28s}", n if n else ">600")

full-batch epochs until training loss < 0.15 (same network, same initial weights):
  SGD lr 0.1                   459
  SGD lr 0.1, momentum 0.9     65
  SGD lr 0.1, Nesterov 0.9     62
  Adagrad lr 0.1               17
  RMSprop lr 0.01              16
  Adam lr 0.01                 36
  AdamW lr 0.01                36

Plain SGD needs 459 epochs; momentum 65; Nesterov 62; Adam and AdamW 36; RMSprop 16; Adagrad 17. Read this carefully rather than as a league table: each optimiser got one hand-picked learning rate, the problem is tiny and easy, and "epochs to reach 0.15" says nothing about how well the network generalises. What it does show is the large, consistent gap between plain SGD and everything that has momentum or per-weight scaling.

6.3Which one should I use?

6.4Chapter summary

• Decay the learning rate: step, exponential, cosine; warm-up first for Adam-style training.
• Measured: plain SGD 459 epochs, momentum 65, Adam 36 on the moons network; Adam is not tuning-free (303 vs 147 steps on the bowl).
• Defaults: AdamW + warm-up + cosine for Transformers; SGD+momentum for classic CNNs; Adam $10^{-3}$ to start.

What is likely to be asked from Week 5, with 30-second answers, deeper explanations, a lecture-or-extra tag on each question, and code you can run.

Week 5 questions are about why each optimiser exists: what problem it fixes, the formula, and the number that shows it. Each answer starts with a 30-second version. Questions are tagged From the notes or Extra.

7.1Interview heat-map: what is most likely

7.2The landscape

degree 1: empirical risk     0.2472   true risk (estimated)       0.4075   gap       0.1604
degree 3: empirical risk     0.0238   true risk (estimated)       0.0551   gap       0.0313
degree 9: empirical risk     0.0000   true risk (estimated)       0.3838   gap       0.3838

batch size   mean of mini-batch gradients   std of the mini-batch gradient   std x sqrt(B)
         1                         -5.848                            8.402           8.40
         8                         -5.898                            3.015           8.53
        64                         -5.947                            1.062           8.49
       512                         -5.962                            0.381           8.61
      4096                         -5.960                            0.140           8.93
same mean (an unbiased estimate), but the noise falls like 1/sqrt(B): that is why a bigger batch tolerates a bigger learning rate

start -2.0  ->  ends at x = -1.3008, f = -3.5139
start -0.5  ->  ends at x = -1.3008, f = -3.5139
start +0.5  ->  ends at x = +1.1309, f = -1.0702
start +2.0  ->  ends at x = +1.1309, f = -1.0702

chord above the curve fails for x^2   : 0 of 100000 random chords (it is convex)
chord above the curve fails for |x|   : 0 (convex, not smooth)
chord above the curve fails for sin(x): 50034 (NOT convex)
chord above the curve fails for x^3   : 50104 (NOT convex on all of R)

gradient at the origin: [0.0, 0.0]  <- zero, yet the origin is NOT a minimum
Hessian eigenvalues   : [-2.0, 2.0]  <- one negative: some direction goes down
start (1, 0.000) exactly on the ridge -> [2e-06, 0.0]  (walks into the saddle and stops)
start (1, 0.001) a hair off the ridge -> [0.0, 56.35]  (slides away, but slowly at first)

 dim   P(all eigenvalues > 0)   (rest are saddles or maxima)
   1     0.4868
   2     0.1547
   3     0.0240
   5     0.0000
   8     0.0000
  12     0.0000
  20     0.0000

f(E[X]) = 3.9946    E[f(X)] = 6.2451
f(E[X]) <= E[f(X)]: True   (the gap is exactly Var(X) = 2.251 )
log: f(E[Y]) = 1.0977 >= E[f(Y)] = 1.0112

7.3Gradient descent and momentum

curvatures 1 and 50, so a stable step needs lr < 2/50 = 0.04
lr = 0.001  : converged in 6821 steps
lr = 0.01   : converged in 679 steps
lr = 0.02   : converged in 338 steps
lr = 0.039  : converged in 199 steps
lr = 0.0399 : converged in 2019 steps
lr = 0.041  : did not converge (diverged)

curvatures 1 and 50, so a stable step needs lr < 2/50 = 0.04
lr = 0.001  : converged in 6821 steps
lr = 0.01   : converged in 679 steps
lr = 0.02   : converged in 338 steps

one Newton step on an ill-conditioned quadratic: [1.  0.3] -> [-0. -0.]  (exact minimum in ONE step)
Newton on the saddle x^2 - y^2 from (1, 1): [0.0, 0.0]  <- lands ON the saddle, not a minimum
d =       1,000 parameters: Hessian has 1e+06 entries = 0.004 GB in float32
d =   1,000,000 parameters: Hessian has 1e+12 entries = 4,000 GB in float32
d = 100,000,000 parameters: Hessian has 1e+16 entries = 40,000,000 GB in float32

  plain GD, best stable lr 0.039   199
  plain GD, lr 0.01                679
  momentum 0.9, lr 0.01            95
  Nesterov 0.9, lr 0.01            62

plain GD is stable for lr < 2/50 = 0.04; heavy-ball momentum for lr < 2(1+beta)/50 = 0.076
momentum 0.9, lr = 0.05: converged in 158 steps
momentum 0.9, lr = 0.07: converged in 149 steps
momentum 0.9, lr = 0.075: converged in 182 steps
momentum 0.9, lr = 0.078: diverged

7.4Adaptive methods and schedules

a constant gradient g = 1 at every step
   step t    accumulated sum    effective lr = lr / sqrt(sum)
        1                1       0.100000
       10               10       0.031623
      100              100       0.010000
     1000             1000       0.003162
    10000            10000       0.001000
the learning rate shrinks like 1/sqrt(t) and never recovers: Adagrad can stall before it finishes
RMSProp after 10000 steps of the same gradient: denominator 1.0 -> effective lr 0.1 (stays near lr)

t=1: m=0.05000  v=0.00025000  m_hat=0.500  v_hat=0.2500  update=0.010000
t=2: m=0.09500  v=0.00049975  m_hat=0.500  v_hat=0.2500  update=0.010000
t=3: m=0.13550  v=0.00074925  m_hat=0.500  v_hat=0.2500  update=0.010000

without bias correction at t=1 the step would be 0.03162 (3.16x too LARGE: v starts at 0 and is badly underestimated, more than m is)
with correction, the very first step is lr * sign(g) = 0.01 : Adam's step size is about lr, whatever the gradient scale
gradient     0.0001: first Adam step 0.009998
gradient     0.5000: first Adam step 0.010000
gradient  5000.0000: first Adam step 0.010000

weights start at [1.0, 1.0], no loss gradient, weight_decay = 0.1, lr = 0.1, 100 steps
decay only, Adam : [0.003, 0.003]  <- L2 sits inside the gradient, so Adam rescales it like any gradient: the weight is driven to ~0
decay only, AdamW: [0.366, 0.366]  <- plain shrinkage factor (1 - lr*wd)^100 = 0.366

opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
warm = torch.optim.lr_scheduler.LinearLR(opt, start_factor=0.01, total_iters=500)   # ramp up
cos = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=10000)                  # then decay
sched = torch.optim.lr_scheduler.SequentialLR(opt, [warm, cos], milestones=[500])
# loop: loss.backward(); opt.step(); sched.step()

epoch   step    exponential   cosine
    0   0.1000   0.1000        0.1000
   15   0.1000   0.0638        0.0946
   30   0.0100   0.0407        0.0794
   60   0.0010   0.0165        0.0345
   90   0.0001   0.0067        0.0024
  100   0.0001   0.0050        0.0000

Week 5 on one page: chapter glance, every optimiser's update rule, the pairs to tell apart, a "which optimiser" guide, a checklist and corrections.

8.1Week 5 at a glance

8.2Every update rule

8.3Key terms

8.4The pairs you must tell apart

8.5Last-minute checklist

8.6Corrections and additions to the source notes