Week 12 Notes

Deep Learning,
Week 12

Introduction to DL and GenAI, Week 12: working with large language models. Why full fine-tuning is out of reach, LoRA with its maths and a working implementation, an experiment comparing it with full fine-tuning, TensorBoard monitoring, retrieval-augmented generation, chunking, embeddings, FAISS exact and approximate search, prompt templates and LangChain-style chains, and a map of the whole course. Every number was really run.

8 chapters11 worked examples~55 min read

LoRAPEFTTensorBoardRAGChunkingEmbeddingsFAISSLCELInterview prep

Chapters

01From Transformer to LLM: the Cost ProblemPart 1
02LoRAPart 2
03LoRA in Practice and TensorBoardPart 3
04Retrieval-Augmented GenerationPart 4
05FAISS, Prompts and ChainsPart 5
06The Whole Course on One MapPart 6
07Interview Question BankInterview
08Week 12 Summary SheetSummary

Part 1

From Transformer to LLM: the Cost Problem

Weeks 9 and 10 built the Transformer; today's large language models (LLMs) are that architecture with larger $d$ , more layers and far more data, pre-trained once on a huge text corpus. This final week asks the practical question: how do you use one for your own task and your own data? There are two main answers. Adapt the model to the task cheaply (fine-tuning, and in particular LoRA), or give the model the facts it lacks at question time (retrieval-augmented generation, RAG).

1.1What you implement to build a Transformer

The notes list four pieces to implement, all met in Week 10 and all present in the small GPT that this week reuses: (1) multi-head self-attention with a causal mask; (2) the position-wise feed-forward network (two linear layers with a ReLU, typically a 4x expansion); (3) residual connections with layer normalisation ("Add & Norm"); (4) stacking the layers. The full data flow of the original model is: tokens $\to$ embedding $+$ positional encoding $\to$ $N$ encoder layers $\to$ encoder output $\to$ $M$ decoder layers (attending to it) $\to$ linear $\to$ softmax over the vocabulary. A decoder-only LLM drops the encoder and the cross-attention.

1.2Why full fine-tuning is out of reach

Full fine-tuning continues training every weight of the pre-trained model on your data. For a 7-billion-parameter model such as LLaMA-7B the weights alone are $7\times10^9\times4$ bytes $=28$ GB in 32-bit floats, and training needs far more than the weights. With the AdamW optimiser every parameter also needs a gradient and two moving averages ( $m$ and $v$ , Week 5). Count it:

main.py

params = 7e9                                              # LLaMA-7B
GB = 1e9
print("full fine-tuning of a 7-billion-parameter model with AdamW")
print(f"  weights   (fp32, 4 bytes)            : {params * 4 / GB:6.0f} GB   <- the notes' 28 GB")
print(f"  gradients (fp32, 4 bytes)            : {params * 4 / GB:6.0f} GB")
print(f"  Adam moments m and v (2 x 4 bytes)   : {params * 8 / GB:6.0f} GB")
print(f"  total before activations             : {params * 16 / GB:6.0f} GB   = 16 bytes per parameter (the notes' 'x16' rule)")
r = 8; layers = 32; lora = layers * 2 * r * (4096 + 4096)  # LoRA r=8 on q and v of every layer
print(f"\nLoRA, r={r} on q and v: {lora:,} trainable parameters ({lora / params:.3%})")
frozen = params * 2 / GB                                   # frozen base weights kept in fp16
trainable = lora * 16 / GB
print(f"  frozen weights (fp16, 2 bytes)       : {frozen:6.1f} GB")
print(f"  trainable parameters x 16 bytes      : {trainable * 1000:6.0f} MB   (weights + gradients + Adam states of only the adapters)")
print(f"  total before activations             : {frozen + trainable:6.1f} GB, about {params * 16 / GB / (frozen + trainable):.0f}x less than full fine-tuning")
print("the saving is about 8x, not 16x: the frozen weights still have to be stored; only gradients and optimiser states shrink to almost nothing")
print("(activations and the batch add more on top of both numbers; quantising the frozen weights, as in QLoRA, shrinks the 14 GB further)")

Output

full fine-tuning of a 7-billion-parameter model with AdamW
  weights   (fp32, 4 bytes)            :     28 GB   <- the notes' 28 GB
  gradients (fp32, 4 bytes)            :     28 GB
  Adam moments m and v (2 x 4 bytes)   :     56 GB
  total before activations             :    112 GB   = 16 bytes per parameter (the notes' 'x16' rule)

LoRA, r=8 on q and v: 4,194,304 trainable parameters (0.060%)
  frozen weights (fp16, 2 bytes)       :   14.0 GB
  trainable parameters x 16 bytes      :     67 MB   (weights + gradients + Adam states of only the adapters)
  total before activations             :   14.1 GB, about 8x less than full fine-tuning
the saving is about 8x, not 16x: the frozen weights still have to be stored; only gradients and optimiser states shrink to almost nothing
(activations and the batch add more on top of both numbers; quantising the frozen weights, as in QLoRA, shrinks the 14 GB further)

Figure 1. Full fine-tuning needs 112 GB before activations; LoRA about 14 GB

Full fine-tuning needs 16 bytes per parameter (4 for the weight, 4 for its gradient, 8 for the two Adam moments), 112 GB for a 7B model before storing a single activation, which is why the notes talk of "150-200 GB, impractical for most": a single GPU rarely has more than 80 GB. Fine-tuning also produces a full separate copy of the model for every task. LoRA, next, removes most of this.

1.3Chapter summary

LLM = a (mostly decoder-only) Transformer pre-trained at scale; to use it for your task you adapt it (fine-tuning, LoRA) or ground it (RAG).
Full fine-tuning of 7B parameters: 28 GB weights + 28 GB gradients + 56 GB Adam states = 112 GB (16 bytes per parameter) before activations.

Next · Part 2LoRA

Part 2

LoRA

LoRA (Low-Rank Adaptation, Hu et al., 2021) rests on an observation: although a weight matrix $W\in\mathbb{R}^{m\times n}$ is large, the change needed to adapt it to a new task is small and can be described by a low-rank matrix. So do not touch $W$ ; learn the change as a product of two thin matrices.

Figure 2. The frozen weight and the trainable low-rank bypass are added

h=Wx+\Delta W\,x=Wx+\frac{\alpha}{r}\,B A\,x,\qquad B\in\mathbb{R}^{m\times r},\ A\in\mathbb{R}^{r\times n},\ r\ll\min(m,n)

$W$ is frozen: the original weights are never modified. (The notes' mistake-list: "LoRA changes the original weights" is false.)
Only $A$ and $B$ are trained: $r(m+n)$ numbers instead of $mn$ .
Initialisation: $A$ random Gaussian, $B=\mathbf{0}$ , so $BA=0$ and the model starts exactly as the pre-trained one. If $B$ were random the very first step would corrupt the model.
Scaling: the update is multiplied by $\alpha/r$ , so changing the rank does not require re-tuning the learning rate.

2.1How many parameters?

main.py

def lora_params(m, n, r): return r * (m + n)                    # B is m x r, A is r x n

print("a single weight matrix W (m x n) against its LoRA update B A with rank r")
print("   m      n      r    full W        LoRA r(m+n)   LoRA / W    reduction")
for m, n, r in ((768, 768, 8), (4096, 4096, 8), (4096, 4096, 16), (4096, 4096, 64), (768, 2304, 8), (4096, 11008, 8)):
    full, lo = m * n, lora_params(m, n, r)
    print(f"{m:5d}  {n:5d}  {r:3d}  {full:11,d}  {lo:11,d}   {lo / full:8.4%}   {1 - lo / full:8.2%}")

# the notes' two examples
print("\nnotes: 768x768, r=8 -> A:", 8 * 768, "+ B:", 768 * 8, "=", lora_params(768, 768, 8), " (", f"{1 - lora_params(768, 768, 8) / 768 ** 2:.1%}", "fewer)")
print("notes: 4096x4096, r=8 -> ", lora_params(4096, 4096, 8), "against", f"{4096 ** 2:,}", f"({1 - lora_params(4096, 4096, 8) / 4096 ** 2:.1%} fewer)")

# a whole model: GPT-2 small adapts the fused q,k,v projection (768 -> 2304) in 12 layers; LLaMA-7B has 32 layers of 4096-wide q and v
print("\nwhole models:")
gpt2 = 12 * lora_params(768, 2304, 8)
print(f"  GPT-2 small  (12 layers, fused c_attn 768x2304, r=8): {gpt2:,} trainable of about 124,000,000 = {gpt2 / 124e6:.2%}")
llama = 32 * 2 * lora_params(4096, 4096, 8)
print(f"  LLaMA-7B     (32 layers, q and v 4096x4096, r=8)    : {llama:,} trainable of about 6,740,000,000 = {llama / 6.74e9:.3%}")
print("the notes guess '0.5-1M for GPT-2'; the count for this common setting is about 0.3M")

Output

a single weight matrix W (m x n) against its LoRA update B A with rank r
   m      n      r    full W        LoRA r(m+n)   LoRA / W    reduction
  768    768    8      589,824       12,288    2.0833%     97.92%
 4096   4096    8   16,777,216       65,536    0.3906%     99.61%
 4096   4096   16   16,777,216      131,072    0.7812%     99.22%
 4096   4096   64   16,777,216      524,288    3.1250%     96.88%
  768   2304    8    1,769,472       24,576    1.3889%     98.61%
 4096  11008    8   45,088,768      120,832    0.2680%     99.73%

notes: 768x768, r=8 -> A: 6144 + B: 6144 = 12288  ( 97.9% fewer)
notes: 4096x4096, r=8 ->  65536 against 16,777,216 (99.6% fewer)

whole models:
  GPT-2 small  (12 layers, fused c_attn 768x2304, r=8): 294,912 trainable of about 124,000,000 = 0.24%
  LLaMA-7B     (32 layers, q and v 4096x4096, r=8)    : 4,194,304 trainable of about 6,740,000,000 = 0.062%
the notes guess '0.5-1M for GPT-2'; the count for this common setting is about 0.3M

Both of the notes' examples check out: a $768\times768$ matrix has 589,824 weights and a rank-8 update needs $6{,}144+6{,}144=12{,}288$ , 97.9% fewer; a $4096\times4096$ matrix (16.7 million) needs 65,536, 99.6% fewer. The ratio is $r(m+n)/mn=r(1/n+1/m)$ , tiny when $m,n$ are large, and it grows linearly with the rank (3.1% at $r=64$ ); typical ranks are 4, 8, 16. For whole models, adapting only the attention projections trains a sliver: 294,912 parameters for GPT-2 small (0.24%) and 4.2 million for LLaMA-7B (0.062%). One correction: the notes guess "0.5-1M for GPT-2" and "100x less memory"; for this common setting the count is about 0.3M and the memory claim needs the careful count of the previous chapter, where full and LoRA training differ by about 8x, not 100x.

2.2LoRA as code

The whole idea is one small module that wraps an nn.Linear. Everything the notes claim can be checked on it: the output is unchanged at the start, only $A$ and $B$ learn, the base is untouched, the update has rank at most $r$ , and the learned update can be merged into $W$ afterwards.

main.py

import torch, torch.nn as nn
torch.manual_seed(0)

class LoRALinear(nn.Module):
    """h = W x + (alpha / r) * B A x, with W frozen."""
    def __init__(self, base: nn.Linear, r: int, alpha: float):
        super().__init__()
        self.base, self.r, self.scale, self.enabled = base, r, alpha / r, True
        for p in base.parameters(): p.requires_grad_(False)                       # the original weights never change
        self.A = nn.Parameter(torch.randn(r, base.in_features) * 0.02)             # A: small random (Gaussian)
        self.B = nn.Parameter(torch.zeros(base.out_features, r))                   # B: zeros, so B A = 0 at the start
    def forward(self, x):
        out = self.base(x)
        return out + self.scale * (x @ self.A.T @ self.B.T) if self.enabled else out
    def merged_weight(self):
        return self.base.weight + self.scale * (self.B @ self.A)

base = nn.Linear(64, 32)
layer = LoRALinear(base, r=4, alpha=8)
x = torch.randn(5, 64)
print("at initialisation B = 0, so the layer equals the original exactly:", torch.equal(layer(x), base(x)))
print("trainable:", [(n, tuple(p.shape)) for n, p in layer.named_parameters() if p.requires_grad], " frozen:", [n for n, p in layer.named_parameters() if not p.requires_grad])
print(f"trainable {sum(p.numel() for p in layer.parameters() if p.requires_grad)} of {sum(p.numel() for p in layer.parameters())} parameters")

target = torch.randn(5, 32)
opt = torch.optim.Adam([p for p in layer.parameters() if p.requires_grad], lr=1e-2)
w_before = base.weight.detach().clone()
for _ in range(200):
    loss = ((layer(x) - target) ** 2).mean(); opt.zero_grad(); loss.backward(); opt.step()
print(f"after training the loss is {loss.item():.4f}; the frozen base weight changed by {(base.weight - w_before).abs().max().item():.1f} (it is untouched)")

merged = nn.Linear(64, 32); merged.weight.data = layer.merged_weight().detach(); merged.bias.data = base.bias.detach()
print("merging W + (alpha/r) B A into one matrix gives the same outputs, with no extra cost at inference:", torch.allclose(merged(x), layer(x), atol=1e-5))
delta = layer.scale * (layer.B @ layer.A)
print("rank of the update matrix (alpha/r) B A:", torch.linalg.matrix_rank(delta, atol=1e-6).item(), "<= r = 4; its size is", tuple(delta.shape))
layer.enabled = False
print("switching the adapter off restores the original model exactly:", torch.equal(layer(x), base(x)))

Output

at initialisation B = 0, so the layer equals the original exactly: True
trainable: [('A', (4, 64)), ('B', (32, 4))]  frozen: ['base.weight', 'base.bias']
trainable 384 of 2464 parameters
after training the loss is 0.1525; the frozen base weight changed by 0.0 (it is untouched)
merging W + (alpha/r) B A into one matrix gives the same outputs, with no extra cost at inference: True
rank of the update matrix (alpha/r) B A: 4 <= r = 4; its size is (32, 64)
switching the adapter off restores the original model exactly: True

Each claim holds. The layer equals the original at initialisation (True), only $A$ ( $4\times64$ ) and $B$ ( $32\times4$ ) receive gradients (384 of 2,464 parameters), the frozen weight changed by exactly 0.0 after training, and the update $(\alpha/r)BA$ has rank 4 even though it is a $32\times64$ matrix. Merging $W+(\alpha/r)BA$ into one matrix gives the same outputs, so LoRA adds no inference cost; and switching the adapter off restores the original model exactly, which is what makes it possible to keep one frozen base model and a small adapter file per task.

2.3Chapter summary

$h=Wx+(\alpha/r)BAx$ ; $W$ frozen, $A\sim\mathcal{N}$ , $B=0$ ; trainable $r(m+n)$ against $mn$ (97.9% and 99.6% fewer for the notes' examples).
Verified: identical output at init, base untouched, update rank $\le r$ , merge gives identical outputs (no inference cost), adapter off restores the base.
Whole-model counts: GPT-2 small 0.24%, LLaMA-7B 0.062% of the weights at $r=8$ .

Next · Part 3LoRA in Practice and TensorBoard

Part 3

LoRA in Practice and TensorBoard

Does LoRA really match full fine-tuning? A small experiment answers it with numbers. Pre-train the tiny GPT of Week 10 on language 1 (the cat chased the fish.), then adapt it to language 2, in which the roles swap (the fish chased the cat.): the same words, a new structure, so the model really must change. Compare full fine-tuning with LoRA at several ranks and on several sets of layers, 300 steps each. Perplexity is on a fixed evaluation set (floor 1.152).

main.py

import copy, math, random, torch, torch.nn as nn
torch.set_num_threads(1); torch.manual_seed(0); random.seed(0)
animals, verbs, things = ["cat", "dog", "bird"], ["saw", "chased", "ate"], ["fish", "ball", "bone"]
def corpus(n, lang):                                         # language 1: animal verb thing.   language 2: THING verb ANIMAL (the roles swap)
    if lang == 1: return "".join(f"the {random.choice(animals)} {random.choice(verbs)} the {random.choice(things)}. " for _ in range(n))
    return "".join(f"the {random.choice(things)} {random.choice(verbs)} the {random.choice(animals)}. " for _ in range(n))
t1, t2 = corpus(1500, 1), corpus(1500, 2)
chars = sorted(set(t1 + t2)); idx = {c: i for i, c in enumerate(chars)}; V = len(chars)
d1, d2 = (torch.tensor([idx[c] for c in t]) for t in (t1, t2))

class LoRALinear(nn.Module):
    def __init__(self, base, r, alpha):
        super().__init__(); self.base, self.scale, self.enabled = base, alpha / r, True
        for p in base.parameters(): p.requires_grad_(False)
        self.A = nn.Parameter(torch.randn(r, base.in_features) * 0.02); self.B = nn.Parameter(torch.zeros(base.out_features, r))
    def forward(self, x):
        out = self.base(x)
        return out + self.scale * (x @ self.A.T @ self.B.T) if self.enabled else out

class Block(nn.Module):                                      # a pre-norm decoder block with explicit q, k, v, o projections
    def __init__(self, d, h):
        super().__init__(); self.h = h; self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.q, self.k, self.v, self.o = (nn.Linear(d, d) for _ in range(4)); self.f1, self.f2 = nn.Linear(d, 4 * d), nn.Linear(4 * d, d)
    def forward(self, x):
        B, T, D = x.shape; y = self.ln1(x)
        q, k, v = [m(y).view(B, T, self.h, D // self.h).transpose(1, 2) for m in (self.q, self.k, self.v)]
        a = nn.functional.scaled_dot_product_attention(q, k, v, is_causal=True).transpose(1, 2).reshape(B, T, D)
        x = x + self.o(a); return x + self.f2(torch.relu(self.f1(self.ln2(x))))
class GPT(nn.Module):
    def __init__(self, d=64, h=4, layers=2, ctx=40):
        super().__init__(); self.tok, self.pos = nn.Embedding(V, d), nn.Embedding(ctx, d)
        self.blocks = nn.ModuleList([Block(d, h) for _ in range(layers)]); self.ln, self.head = nn.LayerNorm(d), nn.Linear(d, V)
    def forward(self, x):
        h = self.tok(x) + self.pos(torch.arange(x.size(1)))
        for b in self.blocks: h = b(h)
        return self.head(self.ln(h))

L = 40
def batch(data, B=64, g=None):
    s = torch.randint(0, len(data) - L - 1, (B,), generator=g); return torch.stack([data[i:i + L] for i in s]), torch.stack([data[i + 1:i + L + 1] for i in s])
EVAL = {k: batch(d, 512, torch.Generator().manual_seed(7)) for k, d in ((1, d1), (2, d2))}      # fixed evaluation batches
def ppl(m, lang):
    with torch.no_grad(): x, y = EVAL[lang]; return math.exp(nn.functional.cross_entropy(m(x).reshape(-1, V), y.reshape(-1)).item())
def train(m, data, steps):
    opt = torch.optim.AdamW([p for p in m.parameters() if p.requires_grad], lr=3e-3)
    for _ in range(steps):
        x, y = batch(data); loss = nn.functional.cross_entropy(m(x).reshape(-1, V), y.reshape(-1)); opt.zero_grad(); loss.backward(); opt.step()
def with_lora(m, r, targets):
    m = copy.deepcopy(m)
    for p in m.parameters(): p.requires_grad_(False)
    for b in m.blocks:
        for name in targets: setattr(b, name, LoRALinear(getattr(b, name), r, 2 * r))
    return m
def adapters(m, on):
    for mod in m.modules():
        if isinstance(mod, LoRALinear): mod.enabled = on

pretrained = GPT(); train(pretrained, d1, 600)                 # "pre-train" on language 1
total = sum(p.numel() for p in pretrained.parameters())
print(f"pre-trained on language 1: perplexity {ppl(pretrained, 1):.3f} on language 1, {ppl(pretrained, 2):.3f} on language 2 (floor 1.152)")
print(f"now adapt to language 2 for 300 steps; the model has {total:,} parameters\n")
print("method                  trainable   share    ppl on L2   adapter OFF: L1 / L2")
for name, make in (("full fine-tuning", lambda: copy.deepcopy(pretrained)), ("LoRA r=4  on q, v", lambda: with_lora(pretrained, 4, ["q", "v"])),
                   ("LoRA r=8  on q, v", lambda: with_lora(pretrained, 8, ["q", "v"])), ("LoRA r=8  on q,k,v,o", lambda: with_lora(pretrained, 8, ["q", "k", "v", "o"])),
                   ("LoRA r=8  on all", lambda: with_lora(pretrained, 8, ["q", "k", "v", "o", "f1", "f2"]))):
    m = make(); n = sum(p.numel() for p in m.parameters() if p.requires_grad)
    train(m, d2, 300); new = ppl(m, 2); adapters(m, False)
    print(f"{name:22s} {n:9,d}  {n / total:6.1%}   {new:8.3f}      {ppl(m, 1):6.3f} / {ppl(m, 2):6.3f}")
print("with the adapter switched off a LoRA model is the original model again (language 1 intact); the fully fine-tuned model has overwritten it")

Output

pre-trained on language 1: perplexity 1.188 on language 1, 5.735 on language 2 (floor 1.152)
now adapt to language 2 for 300 steps; the model has 104,978 parameters

method                  trainable   share    ppl on L2   adapter OFF: L1 / L2
full fine-tuning         104,978  100.0%      1.190       3.717 /  1.190
LoRA r=4  on q, v          2,048    2.0%      1.217       1.188 /  5.735
LoRA r=8  on q, v          4,096    3.9%      1.201       1.188 /  5.735
LoRA r=8  on q,k,v,o       8,192    7.8%      1.193       1.188 /  5.735
LoRA r=8  on all          18,432   17.6%      1.190       1.188 /  5.735
with the adapter switched off a LoRA model is the original model again (language 1 intact); the fully fine-tuned model has overwritten it

Figure 3. LoRA reaches the new language with a fraction of the weights, and the old language survives

Three results. (1) Quality: before adaptation the model scores perplexity 5.74 on language 2. Full fine-tuning reaches 1.190. LoRA with rank 4 on just q and v trains 2,048 weights (2.0%) and reaches 1.217; rank 8 on q,k,v,o (7.8%) reaches 1.193, and rank 8 on every linear layer (17.6%) reaches 1.190, equal to full fine-tuning. More adapted layers and more rank close the gap, as expected. (2) No forgetting: switch the adapter off and a LoRA model returns to exactly the pre-trained model (language 1 perplexity 1.188, language 2 5.735). The fully fine-tuned model has overwritten its weights: its language-1 perplexity went from 1.188 to 3.717. (3) One base, many adapters: because the base is frozen, you can keep one copy of the large model and a tiny adapter per task. All of this on a 105,000-parameter model; at LLM scale the saving in memory and storage is what matters. (A small toy, not a benchmark: on large models LoRA is reported to match full fine-tuning on many tasks, not all.)

3.1Using LoRA with the PEFT library

In practice you do not write the module yourself: HuggingFace's PEFT library wraps a pre-trained model. The recipe is a LoraConfig (rank r, scaling lora_alpha, the layers to adapt as target_modules, dropout) and get_peft_model(model, config), which freezes the base and inserts the $A,B$ pairs. The sketch below is the standard shape (it needs the peft and transformers libraries and a downloaded model, so it is shown, not run here; the module you ran above is what it does inside).

main.py

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained("gpt2")            # the frozen pre-trained model
config = LoraConfig(r=8, lora_alpha=16, target_modules=["c_attn"], lora_dropout=0.05, task_type="CAUSAL_LM")
model = get_peft_model(model, config)                            # freezes W, adds trainable A and B
model.print_trainable_parameters()                               # about 0.3M of ~124M trainable
# ... train as usual with the Trainer or your own loop ...
model.save_pretrained("my-adapter")                              # saves only A and B: a few MB, not the whole model

3.2Watching training: TensorBoard

Fine-tuning runs for a long time, so you log scalars and look at curves. TensorBoard reads event files written by SummaryWriter (the HuggingFace Trainer writes them for you). The metrics the notes name: train/loss, eval/loss (validation loss: the overfitting detector), train/grad_norm (check for explosion), train/learning_rate (the schedule, Week 5), and throughput such as eval/samples_per_second. Here a model is deliberately trained on a tiny dataset (12 sentences) so that it overfits, with the four curves written to disk and then read back exactly as the TensorBoard page does:

main.py

import math, random, tempfile, torch, torch.nn as nn
from torch.utils.tensorboard import SummaryWriter
from tensorboard.backend.event_processing.event_accumulator import EventAccumulator
torch.set_num_threads(1); torch.manual_seed(0); random.seed(0)
animals, verbs, things = ["cat", "dog", "bird"], ["saw", "chased", "ate"], ["fish", "ball", "bone"]
make = lambda n: "".join(f"the {random.choice(animals)} {random.choice(verbs)} the {random.choice(things)}. " for _ in range(n))
train_text, eval_text = make(12), make(400)                        # a TINY training set (12 sentences) invites overfitting
chars = sorted(set(train_text + eval_text)); idx = {c: i for i, c in enumerate(chars)}; V = len(chars)
tr, ev = (torch.tensor([idx[c] for c in t]) for t in (train_text, eval_text))
class Net(nn.Module):
    def __init__(self):
        super().__init__(); self.emb = nn.Embedding(V, 64); self.rnn = nn.LSTM(64, 128, batch_first=True); self.out = nn.Linear(128, V)
    def forward(self, x): return self.out(self.rnn(self.emb(x))[0])
net = Net(); opt = torch.optim.AdamW(net.parameters(), lr=3e-3); L = 30
def batch(data, B=32):
    s = torch.randint(0, len(data) - L - 1, (B,)); return torch.stack([data[i:i + L] for i in s]), torch.stack([data[i + 1:i + L + 1] for i in s])
logdir = tempfile.mkdtemp(); writer = SummaryWriter(logdir)
for step in range(1, 601):
    x, y = batch(tr); loss = nn.functional.cross_entropy(net(x).reshape(-1, V), y.reshape(-1))
    opt.zero_grad(); loss.backward(); gn = nn.utils.clip_grad_norm_(net.parameters(), 1.0); opt.step()
    if step % 20 == 0:
        with torch.no_grad(): ex, ey = batch(ev, 128); el = nn.functional.cross_entropy(net(ex).reshape(-1, V), ey.reshape(-1)).item()
        writer.add_scalar("train/loss", loss.item(), step); writer.add_scalar("eval/loss", el, step)
        writer.add_scalar("train/grad_norm", gn.item(), step); writer.add_scalar("train/learning_rate", opt.param_groups[0]["lr"], step)
writer.close()

acc = EventAccumulator(logdir); acc.Reload()                       # read the logs back exactly as the TensorBoard UI does
print("tags written:", sorted(acc.Tags()["scalars"]))
tr_l = {e.step: e.value for e in acc.Scalars("train/loss")}; ev_l = {e.step: e.value for e in acc.Scalars("eval/loss")}
print("\n step   train/loss   eval/loss")
for s in (20, 60, 100, 200, 400, 600): print(f"{s:5d}    {tr_l[s]:8.3f}    {ev_l[s]:8.3f}")
best = min(ev_l, key=ev_l.get)
print(f"\nlowest eval/loss {ev_l[best]:.3f} at step {best}; by step 600 it is {ev_l[600]:.3f} while train/loss is {tr_l[600]:.3f}")
print("train/loss keeps falling while eval/loss turns upward after step", best, "-> the overfitting signal: stop training (early stopping) or add data/regularisation")
print("largest train/grad_norm logged:", round(max(e.value for e in acc.Scalars('train/grad_norm')), 2), "(a spike here would flag exploding gradients)")

Output

tags written: ['eval/loss', 'train/grad_norm', 'train/learning_rate', 'train/loss']

 step   train/loss   eval/loss
   20       0.985       1.019
   60       0.192       0.279
  100       0.162       0.286
  200       0.124       0.306
  400       0.105       0.374
  600       0.102       0.435

lowest eval/loss 0.261 at step 80; by step 600 it is 0.435 while train/loss is 0.102
train/loss keeps falling while eval/loss turns upward after step 80 -> the overfitting signal: stop training (early stopping) or add data/regularisation
largest train/grad_norm logged: 0.31 (a spike here would flag exploding gradients)

Figure 4. The classic picture: train loss falls forever, eval loss has a minimum and then rises

The training loss falls from 0.985 to 0.102 and never stops improving, while the evaluation loss reaches its lowest value, 0.261 at step 80, and then climbs to 0.435 by step 600: the model is memorising its 12 sentences. This is the notes' rule, "train/loss decreases but eval/loss increases: stop training", in numbers; the standard response is early stopping (keep the checkpoint with the best eval loss), more data or regularisation (Week 4). train/grad_norm stayed below 0.31, no explosion; the learning rate here is constant, and with a schedule the same panel would show its decay. Watching the evaluation curve, not the training one, is the whole point.

3.3Chapter summary

Measured: LoRA r=4 on q,v (2.0% of the weights) reached perplexity 1.217 against 1.190 for full fine-tuning; r=8 on all linear layers (17.6%) matched it (1.190).
Adapter off = the original model (language 1 perplexity 1.188); full fine-tuning forgot it (3.717). One base, many small adapters; the PEFT library does the wrapping.
TensorBoard: watch eval/loss (best 0.261 at step 80, then up to 0.435 while train/loss kept falling: overfitting), train/grad_norm, train/learning_rate.

Next · Part 4Retrieval-Augmented Generation

Part 4

Retrieval-Augmented Generation

Fine-tuning changes how a model behaves; it is a poor way to teach it facts. A language model has three well-known limits: a knowledge cutoff (it knows nothing after its training data was collected), hallucination (it produces fluent, confident, wrong answers, because it is trained to produce likely text, not true text) and no access to private data (your company's documents were never in its training set). Retrieval-augmented generation (RAG) addresses all three by changing the exam from closed-book to open-book: at question time, look up the relevant documents and hand them to the model along with the question.

4.1The architecture

Figure 5. An offline stage builds the index; an online stage retrieves and generates

Document store: the knowledge base (PDFs, pages, tickets), processed once.
Retriever: given a question, finds the most relevant pieces of the store.
Generator (the LLM): reads retrieved context + question and writes the answer. RAG supplements the LLM; it does not replace it: the LLM still does the writing.

The notes' example: "What is the company's Q3 2024 revenue?" An LLM alone has never seen the report and may invent a number. RAG embeds the question, finds the chunk "Q3 2024 report: revenue $5.2B, up 12% year over year", puts it in the prompt, and the model answers from it: grounded, and checkable against the source.

4.2The document pipeline: load, clean, chunk

Steps: load the files and extract text; clean it (headers, page numbers, broken hyphenation); chunk it into pieces; look at chunk statistics (count, length). Chunking is the step that quietly decides retrieval quality, because the unit of retrieval is the chunk. The notes' guidance is 512 to 1024 tokens per chunk with an overlap of 50 to 200 tokens (roughly 10%), for two reasons: too large a chunk drags in irrelevant text and distracts the model; too small a chunk cuts facts in half and loses surrounding context. The overlap exists so that a fact that straddles a cut still appears whole in at least one chunk. A scaled-down experiment (a 179-word document with 12 one-sentence facts, chunks of 30, 60 and 120 words) shows it:

Figure 6. Overlap keeps a fact from being cut in two

main.py

import re
doc = ("Acme Corp reported Q3 2024 revenue of $5.2B, up 12% year over year, driven by cloud subscriptions. "
       "Acme Corp's Q2 2024 revenue was $4.6B with an operating margin of 18%. "
       "The headquarters of Acme Corp is in Pune, India, with regional offices in Berlin and Austin. "
       "Employees at Acme Corp receive 24 days of paid annual leave and 12 days of sick leave. "
       "The remote work policy at Acme Corp allows up to three days of work from home each week. "
       "Acme Corp was founded in 2009 by Meera Rao and Daniel Okafor. "
       "The CEO of Acme Corp is Meera Rao and the CFO is Tomas Lindqvist. "
       "Acme Cloud is the flagship product, offering storage, compute and managed databases. "
       "The Acme support team can be reached at support@acme.example and is open 24 hours on weekdays. "
       "Acme Corp hires about 800 graduates each year through its campus programme. "
       "All Acme Cloud data is encrypted at rest with AES-256 and in transit with TLS 1.3. "
       "The refund policy allows cancelling an Acme Cloud subscription within 30 days for a full refund.")
sentences = [s.strip() for s in re.split(r"(?<=\.)\s+", doc) if s.strip()]
words = doc.split()
def chunk(words, size, overlap):
    step = size - overlap
    return [words[i:i + size] for i in range(0, max(len(words) - overlap, 1), step)]

print(f"the document has {len(words)} words and {len(sentences)} sentences (each sentence is one fact worth retrieving)")
print("\nchunk size  overlap   chunks   facts that sit WHOLLY inside at least one chunk")
longest = max(len(x.split()) for x in sentences)
print(f"the longest fact is {longest} words; an overlap of at least that many words guarantees every fact appears whole in some chunk")
for size in (30, 60, 120):
    for overlap in (0, 8, 16):
        chunks = [" ".join(c) for c in chunk(words, size, overlap)]
        whole = sum(any(s in c for c in chunks) for s in sentences)
        print(f"{size:7d}     {overlap:5d}   {len(chunks):6d}   {whole:2d} of {len(sentences)}")
c0 = [" ".join(c) for c in chunk(words, 30, 0)]
cut = next(s for s in sentences if not any(s in c for c in c0))
print("\nwith 30-word chunks and no overlap this fact is split across two chunks, so neither chunk can answer the question alone:")
print("   ", cut)

Output

the document has 179 words and 12 sentences (each sentence is one fact worth retrieving)

chunk size  overlap   chunks   facts that sit WHOLLY inside at least one chunk
the longest fact is 18 words; an overlap of at least that many words guarantees every fact appears whole in some chunk
     30         0        6    8 of 12
     30         8        8    9 of 12
     30        16       12   12 of 12
     60         0        3   10 of 12
     60         8        4   11 of 12
     60        16        4   12 of 12
    120         0        2   11 of 12
    120         8        2   12 of 12
    120        16        2   12 of 12

with 30-word chunks and no overlap this fact is split across two chunks, so neither chunk can answer the question alone:
    Employees at Acme Corp receive 24 days of paid annual leave and 12 days of sick leave.

With 30-word chunks and no overlap only 8 of the 12 facts survive intact; the leave-policy fact is split across two chunks, so neither chunk answers "how many days of leave?" on its own. An overlap of 16 words (about the length of the longest fact, 18 words, or a little less) makes all 12 survive, at the price of more chunks (12 instead of 6). Larger chunks lose fewer facts but, in a real corpus, mix more unrelated text into each retrieved piece: the trade-off. (Overlap helps but is not monotone for small values: 30 words with an overlap of 8 saved only 9, because shifting the cuts just moves them to other facts. Overlap at least as long as a typical fact is the safe choice.)

4.3Text embeddings and cosine similarity

To find chunks "about the same thing" as a question we turn text into embeddings: dense vectors (384 dimensions for all-MiniLM-L6-v2, 768 for BERT-base, 1536 for text-embedding-ada-002) in which similar meaning means nearby direction (Week 9). Closeness is measured by cosine similarity, $\cos(a,b)=\dfrac{a\cdot b}{\|a\|\,\|b\|}$ , in $[-1,1]$ (1 same direction, 0 orthogonal, $-1$ opposite), which ignores the vectors' lengths. No pre-trained embedding model can be downloaded here, so the demo builds a small one by hand (TF-IDF weighted word counts compressed to 8 dimensions with an SVD, the classical "latent semantic analysis"); the retrieval logic is the same as with a neural embedder.

main.py

import numpy as np
docs = ["Acme Corp reported Q3 2024 revenue of $5.2B, up 12% year over year, driven by cloud subscriptions.",
        "Acme Corp's Q2 2024 revenue was $4.6B with an operating margin of 18%.",
        "The headquarters of Acme Corp is in Pune, India, with regional offices in Berlin and Austin.",
        "Employees at Acme Corp receive 24 days of paid annual leave and 12 days of sick leave.",
        "The remote work policy at Acme Corp allows up to three days of work from home each week.",
        "Acme Corp was founded in 2009 by Meera Rao and Daniel Okafor.",
        "The CEO of Acme Corp is Meera Rao and the CFO is Tomas Lindqvist.",
        "Acme Cloud is the flagship product, offering storage, compute and managed databases.",
        "The Acme support team can be reached at support@acme.example and is open 24 hours on weekdays.",
        "Acme Corp hires about 800 graduates each year through its campus programme.",
        "All Acme Cloud data is encrypted at rest with AES-256 and in transit with TLS 1.3.",
        "The refund policy allows cancelling an Acme Cloud subscription within 30 days for a full refund."]
questions = [("What was the revenue in Q3 2024?", 0), ("Where is the headquarters located?", 2), ("How many days of annual leave do employees get?", 3),
             ("How many days can I work from home?", 4), ("Who founded the company and when?", 5), ("Who is the CEO?", 6),
             ("How many graduates are hired every year?", 9), ("How is customer data encrypted?", 10), ("Can I get a refund after I cancel?", 11), ("What are the support hours?", 8)]
tokenize = lambda s: [w for w in "".join(c.lower() if c.isalnum() else " " for c in s).split() if w not in {"the", "a", "of", "is", "are", "i", "can", "do", "how", "what", "in", "and", "to", "at", "for", "with", "by", "up", "it"}]
vocab = sorted({w for d in docs for w in tokenize(d)}); vi = {w: i for i, w in enumerate(vocab)}

def tfidf(texts, idf):
    M = np.zeros((len(texts), len(vocab)))
    for r, t in enumerate(texts):
        for w in tokenize(t):
            if w in vi: M[r, vi[w]] += 1
    return M * idf
df = np.array([sum(w in tokenize(d) for d in docs) for w in vocab]); idf = np.log((1 + len(docs)) / (1 + df)) + 1
D = tfidf(docs, idf)
U, S, Vt = np.linalg.svd(D, full_matrices=False)
k = 8; proj = Vt[:k].T                                          # a 8-dimensional dense embedding (latent semantic analysis)
embed = lambda texts: (lambda E: E / np.maximum(np.linalg.norm(E, axis=1, keepdims=True), 1e-9))(tfidf(texts, idf) @ proj)
DE, QE = embed(docs), embed([q for q, _ in questions])
cos = QE @ DE.T                                                  # unit vectors: the dot product IS the cosine similarity
v = np.array([1.0, 2.0, 3.0]); cosine = lambda a, b: float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
print(f"cosine(v, 5v) = {cosine(v, 5 * v):.3f} (scale does not matter), cosine(v, -v) = {cosine(v, -v):.3f}, cosine(v, orthogonal) = {cosine(np.array([1.0, 0.0]), np.array([0.0, 1.0])):.3f}; here the question-chunk values lie in [{cos.min():.2f}, {cos.max():.2f}]")
top = np.argsort(-cos, axis=1)
hit1 = np.mean([top[i, 0] == g for i, (_, g) in enumerate(questions)]); hit3 = np.mean([g in top[i, :3] for i, (_, g) in enumerate(questions)])
print(f"retrieval with the SAME embedding for questions and documents: hit@1 = {hit1:.2f}, hit@3 = {hit3:.2f} over {len(questions)} questions")
for i in (0, 2, 7): print(f"  Q: {questions[i][0]:46s} -> best chunk ({cos[i, top[i, 0]]:.2f}): {docs[top[i, 0]][:60]}...")

# WRONG: embed the questions with a different model (a different random projection) than the documents
rng = np.random.default_rng(0); other = rng.normal(size=(len(vocab), k))
QE_bad = tfidf([q for q, _ in questions], idf) @ other; QE_bad /= np.maximum(np.linalg.norm(QE_bad, axis=1, keepdims=True), 1e-9)
topb = np.argsort(-(QE_bad @ DE.T), axis=1)
print(f"questions embedded by a DIFFERENT model than the documents: hit@1 = {np.mean([topb[i, 0] == g for i, (_, g) in enumerate(questions)]):.2f}, hit@3 = {np.mean([g in topb[i, :3] for i, (_, g) in enumerate(questions)]):.2f} (chance is {1 / len(docs):.2f} and {3 / len(docs):.2f})")
print("queries and documents must be embedded with the SAME model, or the vectors live in different spaces")

Output

cosine(v, 5v) = 1.000 (scale does not matter), cosine(v, -v) = -1.000, cosine(v, orthogonal) = 0.000; here the question-chunk values lie in [-0.22, 1.00]
retrieval with the SAME embedding for questions and documents: hit@1 = 0.80, hit@3 = 1.00 over 10 questions
  Q: What was the revenue in Q3 2024?               -> best chunk (0.79): Acme Corp's Q2 2024 revenue was $4.6B with an operating marg...
  Q: How many days of annual leave do employees get? -> best chunk (0.99): Employees at Acme Corp receive 24 days of paid annual leave ...
  Q: How is customer data encrypted?                -> best chunk (1.00): All Acme Cloud data is encrypted at rest with AES-256 and in...
questions embedded by a DIFFERENT model than the documents: hit@1 = 0.10, hit@3 = 0.40 (chance is 0.08 and 0.25)
queries and documents must be embedded with the SAME model, or the vectors live in different spaces

Read the three results. (1) Cosine behaves as defined: $\cos(v,5v)=1$ , $\cos(v,-v)=-1$ , orthogonal $=0$ . (2) With the same embedding for questions and chunks, the right chunk is first for 8 of 10 questions (hit@1 = 0.80) and always within the top 3 (hit@3 = 1.00); the two misses are instructive: "What was the revenue in Q3 2024?" retrieved the Q2 chunk, whose words overlap just as much (this is why you retrieve the top k, not the top 1, and let the LLM read them all). (3) Embed the questions with a different model than the documents and hit@1 collapses to 0.10 (chance is 0.08): the two sets of vectors live in different spaces. This is the notes' "common mistake": queries and documents must use the same embedding model.

4.4Chapter summary

RAG fixes knowledge cutoff, hallucination and private data by retrieving documents and putting them in the prompt: open-book instead of closed-book. The LLM still generates.
Pipeline: load, clean, chunk (512-1024 tokens, overlap 50-200). Measured: 30-word chunks keep 8 of 12 facts with no overlap, 12 of 12 with 16 words of overlap.
Embed with cosine similarity; the same model for queries and documents (hit@1 0.80 with it, 0.10 with different models).

Next · Part 5FAISS, Prompts and Chains

Part 5

FAISS, Prompts and Chains

A real knowledge base has hundreds of thousands of chunks. Comparing the question with every one is a lot of work for each query, which is the job of a vector database: it stores the embeddings and answers "give me the $k$ nearest vectors to this one", fast.

5.1FAISS: exact and approximate search

FAISS (Facebook AI Similarity Search) is the standard library for this. The notes call it "approximate nearest neighbour (ANN) search", and list as a common mistake "FAISS = exact search". The truth is more precise, and worth measuring: FAISS provides both. IndexFlatL2 is an exact brute-force index; IndexIVFFlat (clusters the vectors and searches only the nearest nprobe clusters) and IndexHNSWFlat (a graph) are approximate. The measurement uses 100,000 vectors of 64 dimensions and 200 queries, and scores each index by recall@10: how many of the true 10 nearest neighbours it returned.

main.py

import time, numpy as np, faiss
faiss.omp_set_num_threads(1)
rng = np.random.default_rng(0)
n, d, nq, k = 100_000, 64, 200, 10
centres = rng.normal(size=(200, d)).astype("float32") * 3                   # data with some cluster structure, like real embeddings
xb = (centres[rng.integers(0, 200, n)] + rng.normal(size=(n, d))).astype("float32")
xq = (centres[rng.integers(0, 200, nq)] + rng.normal(size=(nq, d))).astype("float32")

flat = faiss.IndexFlatL2(d); flat.add(xb)                                    # EXACT search: compares the query with every vector
t = time.time(); _, truth = flat.search(xq, k); t_flat = (time.time() - t) / nq * 1000
print(f"{n:,} vectors of dimension {d}; recall@{k} = overlap with the exact answer")
print(f"{'index':26s} {'recall@10':>10s} {'ms / query':>11s}")
print(f"{'IndexFlatL2 (exact)':26s} {1.0:10.3f} {t_flat:11.2f}")
quant = faiss.IndexFlatL2(d); ivf = faiss.IndexIVFFlat(quant, d, 256); ivf.train(xb); ivf.add(xb)       # APPROXIMATE: cluster, then search a few clusters only
for nprobe in (1, 4, 16):
    ivf.nprobe = nprobe; t = time.time(); _, ids = ivf.search(xq, k); ms = (time.time() - t) / nq * 1000
    rec = np.mean([len(set(ids[i]) & set(truth[i])) / k for i in range(nq)])
    print(f"{'IVF, 256 lists, nprobe=' + str(nprobe):26s} {rec:10.3f} {ms:11.2f}")
hnsw = faiss.IndexHNSWFlat(d, 16); hnsw.add(xb); hnsw.hnsw.efSearch = 32
t = time.time(); _, ids = hnsw.search(xq, k); ms = (time.time() - t) / nq * 1000
print(f"{'HNSW graph (M=16)':26s} {np.mean([len(set(ids[i]) & set(truth[i])) / k for i in range(nq)]):10.3f} {ms:11.2f}")
print("FAISS offers BOTH: an exact index (Flat) and approximate ones; approximate ones trade a little recall for a large speed-up")

Output

100,000 vectors of dimension 64; recall@10 = overlap with the exact answer
index                       recall@10  ms / query
IndexFlatL2 (exact)             1.000        0.69
IVF, 256 lists, nprobe=1        0.899        0.01
IVF, 256 lists, nprobe=4        1.000        0.03
IVF, 256 lists, nprobe=16       1.000        0.10
HNSW graph (M=16)               0.942        0.05
FAISS offers BOTH: an exact index (Flat) and approximate ones; approximate ones trade a little recall for a large speed-up

Figure 7. Approximate indexes answer much faster at little or no loss of recall

The exact index takes 0.69 ms per query (recall 1.000 by definition). The IVF index with nprobe=1 takes 0.01 ms, about 70 times faster, at recall 0.899: it sometimes misses a true neighbour because it looked in only one cluster. Raising nprobe to 4 restores recall to 1.000 and is still about 20 times faster; HNSW gives recall 0.942 at 0.05 ms. That is the trade-off in the notes: "slight accuracy loss for a massive speed gain". (Honest limits: this data is cleanly clustered, so recall recovers easily; harder data needs more probing, and with a few thousand chunks, as in a small RAG system, an exact index is perfectly fine.)

5.2The online flow and the prompt

The online path is: embed the query with the same model, search the index for the top $k$ chunks, fill a prompt template with them, and let the LLM answer. The prompt template is the part the notes call critical: it should (1) present context then question, (2) tell the model to use only the provided context, and (3) tell it to say "I don't know" if the answer is not there, which is what prevents it from inventing one when retrieval fails.

5.3LangChain and the "|" chain

Figure 8. Each stage is a Runnable; the pipe sends the output of one into the next

LangChain packages these steps and its Expression Language (LCEL) joins them with the pipe operator: chain = retriever | prompt | llm | parser. The idea is plain function composition: every component is a Runnable with an invoke method, and a | b means "feed the output of a to b". LangChain is not installed here, so the demo implements that composition in ten lines, with a stand-in retriever and a stub that plays the LLM (it only obeys the instruction in the prompt; it is not a language model). It shows the data flow faithfully, including the failure case:

main.py

import re
# A tiny stand-in for LangChain's "|" composition (LCEL): every step is a Runnable, and a | b feeds the output of a into b.
class Runnable:
    def __init__(self, fn): self.fn = fn
    def invoke(self, x): return self.fn(x)
    def __or__(self, other): return Runnable(lambda x: other.invoke(self.invoke(x)))

docs = ["Acme Corp reported Q3 2024 revenue of $5.2B, up 12% year over year.",
        "The headquarters of Acme Corp is in Pune, India.",
        "Employees at Acme Corp receive 24 days of paid annual leave.",
        "All Acme Cloud data is encrypted at rest with AES-256 and in transit with TLS 1.3."]
words = lambda s: set(re.findall(r"[a-z0-9$.]+", s.lower())) - {"the", "a", "of", "is", "in", "at", "what", "how", "was", "are", "do", "and", "with"}
def retrieve(q, k=1):                                      # stand-in retriever: word overlap instead of embeddings
    scored = sorted(((len(words(q) & words(d)), d) for d in docs), reverse=True)[:k]
    return [d for s, d in scored if s > 0]
TEMPLATE = "Use ONLY the context to answer. If the answer is not in the context, say \"I don't know\".\nContext: {context}\nQuestion: {question}\nAnswer:"
prompt = Runnable(lambda p: TEMPLATE.format(context=" ".join(p["docs"]), question=p["question"]))
def stub_llm(text):                                        # NOT a real language model: it only obeys the instruction in the prompt
    context = re.search(r"Context: (.*)\nQuestion", text, re.S).group(1).strip(); question = re.search(r"Question: (.*)\nAnswer", text).group(1)
    return f"Based on the context: {context}" if context else "I don't know"
chain = Runnable(lambda q: {"question": q, "docs": retrieve(q)}) | prompt | Runnable(stub_llm) | Runnable(str.strip)    # retriever | prompt | LLM | parser

for q in ("What was the Q3 2024 revenue?", "How is data encrypted?", "What is the price of the premium plan?"):
    print("Q:", q); print("A:", chain.invoke(q)); print()
print("the prompt that the LLM would actually receive for the first question:")
print(prompt.invoke({"question": "What was the Q3 2024 revenue?", "docs": retrieve("What was the Q3 2024 revenue?")}))
print("\n(the LLM here is a stub; a real RAG system sends this prompt to a model such as an open-source LLM or an API)")

Output

Q: What was the Q3 2024 revenue?
A: Based on the context: Acme Corp reported Q3 2024 revenue of $5.2B, up 12% year over year.

Q: How is data encrypted?
A: Based on the context: All Acme Cloud data is encrypted at rest with AES-256 and in transit with TLS 1.3.

Q: What is the price of the premium plan?
A: I don't know

the prompt that the LLM would actually receive for the first question:
Use ONLY the context to answer. If the answer is not in the context, say "I don't know".
Context: Acme Corp reported Q3 2024 revenue of $5.2B, up 12% year over year.
Question: What was the Q3 2024 revenue?
Answer:

(the LLM here is a stub; a real RAG system sends this prompt to a model such as an open-source LLM or an API)

The first two questions are answered from the retrieved chunk. The third asks for something that is in no document, so retrieval returns nothing and the prompt's instruction ("if the answer is not in the context, say I don't know") yields "I don't know" instead of an invented price: that is how RAG reduces hallucination. With a real LLM the same prompt is sent to the model, which is much better at paraphrasing and combining chunks than this stub, though it can still ignore the instruction, so the prompt reduces hallucination but does not eliminate it.

5.4Checking a RAG system

Retrieval quality check (the notes): take questions whose answer you know and check that the relevant chunk appears in the top 3. If not, change the chunk size, the embedding model or $k$ . The demo's hit@3 of 1.00 is exactly this check.
Separate retrieval failures (the right chunk was not retrieved) from generation failures (it was retrieved but the model ignored or misread it): they have different fixes.
Keep the knowledge base current: updating RAG means re-indexing documents, not retraining the model, which is its main practical advantage over fine-tuning for facts.

5.5Chapter summary

FAISS offers exact (Flat) and approximate (IVF, HNSW) indexes. Measured on 100,000 vectors: IVF nprobe=1 was 70x faster at recall 0.899; nprobe=4 reached recall 1.000 about 20x faster. The notes' "FAISS = exact" mistake-list entry is half the story.
Online flow: embed (same model), top-k search, fill the prompt (context, question, "use only the context, else I don't know"), LLM answers.
LCEL: retriever | prompt | llm | parser is function composition; a demo with a stub LLM showed a grounded answer and an honest "I don't know".

Next · Part 6The Whole Course on One Map

Part 6

The Whole Course on One Map

You have now met every topic of the course. This closing chapter does not add anything new; it shows how the twelve weeks hang together, because the end-term rewards seeing the same ideas return in different clothes.

Figure 9. Twelve weeks, eight blocks

6.1Ideas that keep coming back

Idea	Where it appears	The connection
Vanishing and exploding gradients	sigmoid (W1), optimisation (W5), BPTT (W7), deep Transformers (W10)	everything is about gradients flowing backwards
Skip / residual connections	ResNet add and U-Net concatenate (W4), LSTM cell state (W7), Transformer (W10), diffusion U-Net (W11)	give the gradient a direct path in deep networks
Encoder, compress, decoder	U-Net (W4), seq2seq (W8), Transformer (W10), VAE and LDM (W11)	compress, process, expand
Attention	fixes the seq2seq bottleneck (W9), Transformer (W10), cross-attention in latent diffusion (W11), retrieval in RAG (W12)	the core of modern deep learning
Sinusoidal embeddings	Transformer positions (W10), diffusion step (W11)	same formula, different purpose
Normalisation	BatchNorm (W4), LayerNorm in Transformers (W10), diffusion and LLMs (W11, W12)	keeps activations in a range that trains well
Transfer learning	pre-trained CNN (W4), fine-tuned BERT (W10), LoRA (W12)	a progression towards cheaper adaptation
Loss functions	MSE (W1), cross-entropy (W1, W7), GAN minimax (W6), diffusion MSE on noise (W11)	the loss follows the task

6.2What the measurements taught us

Across the course the notes' claims were checked by running code, and a handful turned out to need care. The ones worth remembering are the kind that decide marks: a default-initialised LSTM does not beat an RNN; its forget gate must start near 1 (W7). An encoder layer has $12d^2$ parameters, not $8d^2$ (W10). "31.6% of the image remains" is a coefficient, not a percentage (W11). FAISS offers exact and approximate search (W12). DDIM and beam search trade quality for speed in measurable steps (W8, W11). And a point that never changes: the training loss going down proves nothing, only the held-out loss does (W4, W12).

6.3A study order for the end term

Derive and write by hand: backprop for a small network (W1), the CNN output-size formula and parameter count (W3), the Adam update (W5), the GAN objective and $D^*$ (W6), the LSTM equations (W7), $\mathrm{softmax}(QK^\top/\sqrt{d_k})V$ and multi-head attention (W9), the forward closed form and the noise-prediction loss (W11), LoRA's $r(m+n)$ (W12).
Compare in tables: BERT against GPT, DDPM against DDIM, classifier against classifier-free guidance, GRU against LSTM, LayerNorm against BatchNorm, RAG against fine-tuning.
Know the numbers that anchor each topic (the summary sheet of each week lists them): 1.386 and 2 log 2 for the GAN, 12 $d^2$ per layer, 0.316 versus 0.10 for the diffusion example, 16 bytes per parameter for Adam.
Last, re-read each week's summary sheet and its corrections table: the corrections are the places where an unwary answer loses marks.

Next · InterviewInterview Question Bank

Interview

Interview Question Bank

What is likely to be asked from Week 12, with 30-second answers, deeper explanations, a lecture-or-extra tag on each question, and code you can run.

Week 12 questions are about adapting an LLM cheaply (LoRA), grounding it with documents (RAG), and the pieces in between (embeddings, chunking, FAISS, prompts). Each answer starts with a 30-second version. Questions are tagged From the notes or Extra.

1Definition

One clear sentence: what it is.

2Intuition or tiny example

Why it works, in one picture or three lines.

3The failure it prevents

What goes wrong without it.

7.1Interview heat-map: what is most likely

Very likely5 topics

LoRA concept and math"Explain LoRA. How does it reduce parameters?"Notes 2, 3, Ex. 1

LoRA initialisation and what stays frozen"Why is B initialised to zero? Does LoRA change the original weights?"Notes 2, 7

RAG architecture"Describe the RAG architecture and the problems it solves."Notes 2, 8

Embeddings and cosine similarity"How does retrieval work? Why the same embedding model?"Notes 2, 3, 7

Document chunking"How should documents be chunked, and why overlap?"Notes 2, Ex. 3

Likely5 topics

FAISS and approximate nearest neighbours"What is FAISS? Exact or approximate?"Notes 2, 7

Prompt template design"Why is the RAG prompt template important?"Notes 2

Memory of full fine-tuning"Estimate the memory to fine-tune a 7B model with and without LoRA."Notes 2, 6

TensorBoard and overfitting"Which curves do you watch while fine-tuning an LLM?"Notes 2

RAG versus fine-tuning"When RAG and when fine-tuning?"Notes 2, Ex. 2

Sometimes1 topics

LangChain LCEL"What is LCEL?"Notes 2

7.2LoRA

Interview 1Explain LoRA, write its equation and show how it reduces the number of trainable parameters.

Week 12Very likelyNotes 2, 3, Ex. 1From the lecture

Freeze W; learn a low-rank update B A.

Prints

a single weight matrix W (m x n) against its LoRA update B A with rank r
   m      n      r    full W        LoRA r(m+n)   LoRA / W    reduction
  768    768    8      589,824       12,288    2.0833%     97.92%
 4096   4096    8   16,777,216       65,536    0.3906%     99.61%
 4096   4096   16   16,777,216      131,072    0.7812%     99.22%

Prints

at initialisation B = 0, so the layer equals the original exactly: True
trainable: [('A', (4, 64)), ('B', (32, 4))]  frozen: ['base.weight', 'base.bias']
trainable 384 of 2464 parameters
after training the loss is 0.1525; the frozen base weight changed by 0.0 (it is untouched)

Answer

Say this first (30 seconds)

Full fine-tuning updates every weight (7B parameters: 112 GB of training state). LoRA freezes the pre-trained $W\in\mathbb{R}^{m\times n}$ and learns the update as a product of two thin matrices: $h=Wx+\tfrac{\alpha}{r}BAx$ , $B\in\mathbb{R}^{m\times r}$ , $A\in\mathbb{R}^{r\times n}$ , $r\ll\min(m,n)$ . Only $A$ and $B$ are trained: $r(m+n)$ parameters instead of $mn$ , a ratio $r(1/m+1/n)$ . Example: $768\times768$ , $r=8$ : $12{,}288$ against $589{,}824$ , 97.9% fewer; $4096\times4096$ : 65,536 against 16.7M, 99.6% fewer. Init: $A$ random Gaussian, $B=0$ , so training starts from the pre-trained model; the update is scaled by $\alpha/r$ .

Go deeper

Verified in code: identical output at init, base weight unchanged after training, update rank at most r, merge gives identical outputs.

Follow-up they may askWhat happens to the parameter count if r doubles?

Interview 2Why is $B$ initialised to zero and $A$ randomly? Does LoRA change the original weights?

Week 12Very likelyNotes 2, 7From the lecture

So that BA = 0 at the start.

Prints

method                  trainable   share    ppl on L2   adapter OFF: L1 / L2
full fine-tuning         104,978  100.0%      1.190       3.717 /  1.190
LoRA r=4  on q, v          2,048    2.0%      1.217       1.188 /  5.735
LoRA r=8  on q, v          4,096    3.9%      1.201       1.188 /  5.735
LoRA r=8  on q,k,v,o       8,192    7.8%      1.193       1.188 /  5.735
LoRA r=8  on all          18,432   17.6%      1.190       1.188 /  5.735
with the adapter switched off a LoRA model is the original model again (language 1 intact); the fully fine-tuned model has overwritten it

Answer

Say this first (30 seconds)

With $B=0$ the update $BA=0$ , so the adapted model equals the pre-trained model at step 0 and training starts from a good place; if $B$ were random the first step would disturb the model. $A$ is random so that the gradients of $B$ are not zero (with both zero, nothing would learn). LoRA never changes $W$ : the original weights stay frozen and only $A$ and $B$ are trained; after training they can be merged, $W+\tfrac{\alpha}{r}BA$ , or the adapter can be dropped to recover the original model.

Go deeper

Measured: adapter off gave the original language-1 perplexity (1.188); the fully fine-tuned model had forgotten it (3.717).

Follow-up they may askWhat if both A and B start at zero?

Interview 3Estimate the memory needed to fully fine-tune a 7B-parameter model with AdamW, and with LoRA.

Week 12LikelyNotes 2, 6From the lecture

16 bytes per parameter, versus mostly frozen weights.

Prints

full fine-tuning of a 7-billion-parameter model with AdamW
  weights   (fp32, 4 bytes)            :     28 GB   <- the notes' 28 GB
  gradients (fp32, 4 bytes)            :     28 GB
  Adam moments m and v (2 x 4 bytes)   :     56 GB
  total before activations             :    112 GB   = 16 bytes per parameter (the notes' 'x16' rule)

LoRA, r=8 on q and v: 4,194,304 trainable parameters (0.060%)
  frozen weights (fp16, 2 bytes)       :   14.0 GB
  trainable parameters x 16 bytes      :     67 MB   (weights + gradients + Adam states of only the adapters)
  total before activations             :   14.1 GB, about 8x less than full fine-tuning
the saving is about 8x, not 16x: the frozen weights still have to be stored; only gradients and optimiser states shrink to almost nothing
(activations and the batch add more on top of both numbers; quantising the frozen weights, as in QLoRA, shrinks the 14 GB further)

Answer

Say this first (30 seconds)

Full: weights 4 bytes + gradient 4 + Adam $m$ and $v$ 8 = 16 bytes per parameter, $7\times10^9\times16=112$ GB before activations. LoRA ( $r=8$ on $q,v$ , 4.2M trainable): frozen fp16 weights 14 GB + about 67 MB of adapter, gradients and optimiser state $\approx14$ GB, about 8x less, and the weights can be quantised further. The notes' "~16x less" is too generous: the frozen weights still have to be held; it is the gradients and optimiser states that nearly vanish.

Go deeper

Counted: 112 GB against 14.1 GB.

Follow-up they may askWhat does QLoRA add?

Interview 4Which metrics do you monitor in TensorBoard while fine-tuning, and how do you detect overfitting?

Week 12LikelyNotes 2Extra

eval/loss against train/loss.

Prints

   20       0.985       1.019
   60       0.192       0.279
  100       0.162       0.286
  200       0.124       0.306
  400       0.105       0.374
  600       0.102       0.435

lowest eval/loss 0.261 at step 80; by step 600 it is 0.435 while train/loss is 0.102
train/loss keeps falling while eval/loss turns upward after step 80 -> the overfitting signal: stop training (early stopping) or add data/regularisation

Answer

Say this first (30 seconds)

train/loss, eval/loss (validation loss), train/grad_norm (a spike means exploding gradients), train/learning_rate (the schedule) and throughput such as eval/samples_per_second. Overfitting: train/loss keeps decreasing while eval/loss rises; stop (early stopping) at the lowest eval loss.

Go deeper

Measured: train/loss 0.985 to 0.102 while eval/loss was best at 0.261 (step 80) and rose to 0.435.

Follow-up they may askWhich curve do you trust, train or eval, and why?

7.3RAG

Interview 5Describe the RAG architecture and data flow. What problems does it solve that an LLM alone cannot?

Week 12Very likelyNotes 2, 8From the lecture

Open-book exam: retrieve, then generate.

Prints

Q: What was the Q3 2024 revenue?
A: Based on the context: Acme Corp reported Q3 2024 revenue of $5.2B, up 12% year over year.

Q: How is data encrypted?
A: Based on the context: All Acme Cloud data is encrypted at rest with AES-256 and in transit with TLS 1.3.

Q: What is the price of the premium plan?
A: I don't know

Answer

Say this first (30 seconds)

Components: a document store (knowledge base), a retriever, and a generator (the LLM). Offline: documents $\to$ clean $\to$ chunk $\to$ embed $\to$ store in a vector database (FAISS). Online: question $\to$ embed (same model) $\to$ top- $k$ search $\to$ fill the prompt with context and question $\to$ LLM answers. Solves: knowledge cutoff (outdated information), hallucination (answers grounded in retrieved facts), and private/proprietary data. It supplements the LLM, which still writes the answer.

Go deeper

Demonstrated: a grounded answer when a fact was retrieved and "I don't know" when nothing relevant existed.

Follow-up they may askIs RAG a replacement for fine-tuning?

Interview 6How does a RAG retriever find relevant chunks, and why must queries and documents use the same embedding model?

Week 12Very likelyNotes 2, 3, 7From the lecture

Nearest vectors by cosine similarity.

Prints

cosine(v, 5v) = 1.000 (scale does not matter), cosine(v, -v) = -1.000, cosine(v, orthogonal) = 0.000; here the question-chunk values lie in [-0.22, 1.00]
retrieval with the SAME embedding for questions and documents: hit@1 = 0.80, hit@3 = 1.00 over 10 questions
  Q: What was the revenue in Q3 2024?               -> best chunk (0.79): Acme Corp's Q2 2024 revenue was $4.6B with an operating marg...
  Q: How many days of annual leave do employees get? -> best chunk (0.99): Employees at Acme Corp receive 24 days of paid annual leave ...
  Q: How is customer data encrypted?                -> best chunk (1.00): All Acme Cloud data is encrypted at rest with AES-256 and in...
questions embedded by a DIFFERENT model than the documents: hit@1 = 0.10, hit@3 = 0.40 (chance is 0.08 and 0.25)
queries and documents must be embedded with the SAME model, or the vectors live in different spaces

Answer

Say this first (30 seconds)

Each chunk is mapped to a dense embedding (e.g. 384-d for all-MiniLM-L6-v2, 768-d BERT-base, 1536-d ada-002) so that similar meaning means nearby directions. The query is embedded and the chunks with the highest cosine similarity $\tfrac{a\cdot b}{\|a\|\|b\|}\in[-1,1]$ are returned (top- $k$ ). The same model must embed both, because different models place texts in different vector spaces, so their distances mean nothing to each other.

Go deeper

Measured: hit@1 0.80 and hit@3 1.00 with one embedding; hit@1 0.10 when queries used a different model.

Follow-up they may askWhy retrieve top-3 rather than only the top-1?

Interview 7Describe the document-processing pipeline. How do you choose the chunk size and overlap?

Week 12Very likelyNotes 2, Ex. 3From the lecture

Not too large, not too small, with overlap.

Prints

chunk size  overlap   chunks   facts that sit WHOLLY inside at least one chunk
the longest fact is 18 words; an overlap of at least that many words guarantees every fact appears whole in some chunk
     30         0        6    8 of 12
     30         8        8    9 of 12
     30        16       12   12 of 12
     60         0        3   10 of 12
     60         8        4   11 of 12
     60        16        4   12 of 12
    120         0        2   11 of 12
    120         8        2   12 of 12
    120        16        2   12 of 12

with 30-word chunks and no overlap this fact is split across two chunks, so neither chunk can answer the question alone:

Answer

Say this first (30 seconds)

Load files and extract text, clean artefacts, chunk, then analyse chunk statistics. Typical chunk 512 to 1024 tokens, overlap 50 to 200 (about 10%). Too large: few chunks, lots of irrelevant text, the LLM is distracted. Too small: facts cut mid-sentence, context lost. Overlap makes a fact straddling a boundary appear whole in at least one chunk; without it boundary information is lost. Tune against a retrieval check (the known answer should be in the top 3).

Go deeper

Measured: 30-word chunks without overlap kept 8 of 12 facts whole; with 16 words of overlap all 12.

Follow-up they may askWhy must the overlap be at least as long as a typical fact?

Interview 8What should a good RAG prompt template contain and why?

Week 12LikelyNotes 2From the lecture

Context, question, and an instruction to stay grounded.

Prints

the prompt that the LLM would actually receive for the first question:
Use ONLY the context to answer. If the answer is not in the context, say "I don't know".
Context: Acme Corp reported Q3 2024 revenue of $5.2B, up 12% year over year.
Question: What was the Q3 2024 revenue?
Answer:

Answer

Say this first (30 seconds)

Three ingredients: (1) the retrieved context followed by the question, so the model answers from what it is shown; (2) an instruction to use only the provided context; (3) an instruction to say "I don't know" if the answer is not in the context. (2) and (3) are what stop the model from falling back on its memory and hallucinating when retrieval fails. They reduce, but do not eliminate, hallucination.

Go deeper

Demonstrated: with nothing retrieved, the chain returned "I don't know" instead of an invented price.

Follow-up they may askWhat goes wrong if the context is placed after the question and not delimited?

7.4Tools and choices

Interview 9What is FAISS? Is its search exact or approximate? Why approximate?

Week 12LikelyNotes 2, 7From the lecture

A vector-search library with both kinds of index.

Prints

100,000 vectors of dimension 64; recall@10 = overlap with the exact answer
index                       recall@10  ms / query
IndexFlatL2 (exact)             1.000        0.69
IVF, 256 lists, nprobe=1        0.899        0.01
IVF, 256 lists, nprobe=4        1.000        0.03
IVF, 256 lists, nprobe=16       1.000        0.10
HNSW graph (M=16)               0.942        0.05
FAISS offers BOTH: an exact index (Flat) and approximate ones; approximate ones trade a little recall for a large speed-up

Answer

Say this first (30 seconds)

FAISS is a library for efficient similarity search over dense vectors, the engine of the vector database. Exact search compares the query with every vector (cost grows with the database). Approximate nearest neighbour (ANN) indexes (IVF clusters, HNSW graphs, product quantisation) look at only a promising part, trading a little recall for a large speed-up. The notes call FAISS approximate; precisely, it offers both (IndexFlatL2 is exact) and the approximate ones are what make million-vector search fast.

Go deeper

Measured on 100k vectors: IVF nprobe=1 was 70x faster at recall 0.899; nprobe=4 had recall 1.000 and was about 20x faster.

Follow-up they may askWhat does nprobe control?

Interview 10What is LangChain Expression Language (LCEL)? Write a RAG chain.

Week 12SometimesNotes 2Extra

The pipe composes Runnables.

Prints

Q: What was the Q3 2024 revenue?
A: Based on the context: Acme Corp reported Q3 2024 revenue of $5.2B, up 12% year over year.

Q: How is data encrypted?

Answer

Say this first (30 seconds)

LCEL composes components with the | operator: every component is a Runnable and a | b sends the output of a to b. A RAG chain is retriever | prompt_template | LLM | output_parser: retrieve chunks, fill the prompt, generate, turn the model output into a string. It is plain function composition with conveniences such as streaming and batching.

Go deeper

Reproduced the composition in ten lines with a stub LLM.

Follow-up they may askWhere does the retriever get its documents from?

Interview 11When would you choose RAG, when LoRA fine-tuning, and when both?

Week 12LikelyNotes 2, Ex. 2Extra

Facts or behaviour?

Answer

Say this first (30 seconds)

RAG for knowledge: fresh, private or large fact collections that change; updating means re-indexing, not retraining, and the source can be cited. LoRA fine-tuning for behaviour: a style, a format, a domain language, a task skill (Week 12's toy changed how the model writes, not what it knows). Both together are common: a fine-tuned model that follows your format, fed with retrieved facts. RAG does not change the model weights; LoRA does not add new knowledge reliably.

Go deeper

Rule of thumb, not a law.

Follow-up they may askWhich one lets you show the user the source of an answer?

Next · SummaryWeek 12 Summary Sheet

Summary

Week 12 Summary Sheet

Week 12 on one page: chapter glance, formulas, the RAG pipeline, a LoRA-versus-RAG guide, a checklist and corrections to the source notes. The last section of Chapter 6 gives a study order for the whole course.

8.1Week 12 at a glance

C1The problem

Full fine-tuning of 7B: 28 + 28 + 56 = 112 GB (16 bytes per parameter) before activations.

C2LoRA

$h=Wx+(\alpha/r)BAx$ ; $W$ frozen; $A$ random, $B=0$ ; $r(m+n)$ parameters. 768x768, r=8: 97.9% fewer; 4096x4096: 99.6%.

C3In practice

Toy: LoRA r=4 on q,v (2.0%) ppl 1.217 against full 1.190; adapter off keeps language 1 (1.188 against 3.717). TensorBoard: watch eval/loss.

C4RAG

Retrieve then generate (open book): fixes cutoff, hallucination, private data. Chunk 512-1024 tokens, overlap 50-200. Same embedding model; cosine similarity.

C5FAISS and chains

FAISS: exact (Flat) and approximate (IVF, HNSW). Prompt: context, question, "only the context, else I don't know". LCEL: retriever | prompt | llm | parser.

C6The whole course

Gradients, skip connections, attention, encoder-decoder, transfer learning keep returning.

8.2Formulas and numbers

Item	Formula	Number
LoRA forward	$h=Wx+\tfrac{\alpha}{r}BAx$	$B=0$ at init
LoRA parameters	$r(m+n)$ vs $mn$	ratio $r(1/m+1/n)$
Full fine-tuning memory	16 bytes per parameter	7B: 112 GB
LoRA memory (7B, r=8)	frozen fp16 + adapters	14.1 GB (about 8x less)
Cosine	$\tfrac{a\cdot b}{\\|a\\|\\|b\\|}$	$[-1,1]$
Embedding sizes	MiniLM / BERT-base / ada-002	384 / 768 / 1536
Chunk / overlap	tokens	512-1024 / 50-200

8.3RAG against fine-tuning

	RAG	LoRA fine-tuning
Changes	the context given to the model	a few adapter weights
Good for	facts: fresh, private, large	behaviour: style, format, skill
To update	re-index documents	train again
Cites sources	yes	no

8.4Key terms

C1-C3

LoRA: low-rank adapter on frozen weights.
PEFT: library for parameter-efficient fine-tuning.
rank r: width of the update.
TensorBoard: training curves.

C4-C5

RAG: retrieval-augmented generation.
chunk: unit of retrieval.
FAISS: vector search library.
LCEL: pipe composition.

8.5The pairs you must tell apart

LoRA $W$ frozen

LoRA A, Btrained

Bzeros at init

Arandom Gaussian

RAGadds knowledge at question time

fine-tuningchanges behaviour / weights

exact indexIndexFlat

approximate indexIVF, HNSW

queries and documentsSAME embedding model

different modelsdifferent spaces

train/loss falling, eval/loss risingoverfitting

both fallingstill learning

8.6Last-minute checklist

C1-C3

LoRA equation and $r(m+n)$
$B=0$ , $W$ frozen
97.9% and 99.6% examples

C4-C5

RAG: offline index, online retrieve + generate
same embedding model
chunk with overlap
"only the context, else I don't know"

8.7Corrections and additions to the source notes

Where	The source says	Better
FAISS	"FAISS = approximate; exact is a mistake"	FAISS provides both: IndexFlatL2 is exact, IVF and HNSW are approximate (measured recall 1.000, 0.899 to 1.000, 0.942)
LoRA memory	"about 16x less memory"; "GPT-2: 0.5-1M trainable, 100x less memory"	7B: 112 GB against 14.1 GB (about 8x) because the frozen weights are still stored; GPT-2 small at r=8 trains about 0.29M
Full fine-tuning memory	"~150-200 GB"	16 bytes per parameter gives 112 GB before activations; activations push the total up toward the notes' range
LoRA hypothesis	"weight updates are low-rank"	an empirical observation that holds for many tasks, not a theorem; the toy needed rank 8 on all layers to match full fine-tuning
Chunking	"optimal 512-1024 tokens with ~10% overlap"	a heuristic: tune it per corpus with a retrieval check; in the demo a small overlap helped unevenly while one at least as long as a fact fixed it
RAG	"eliminates hallucination"	it reduces it; the model can still misread or ignore the context, and retrieval can miss (hit@1 was 0.80)

Revision overview

The problem112 GB for 7B
16 bytes / parameter
a copy per task

LoRAW frozen
B A, r(m+n)
B = 0 at init

MonitorTensorBoard
eval/loss
early stop

Week 12Adapt and ground an LLM

RAGopen book
retrieve + generate
cutoff, hallucination

Indexchunk + overlap
embed (same model)
FAISS flat / IVF / HNSW

Chainretriever | prompt
| llm | parser
only the context

Revision: all of Week 12 on one page

Deep Learning,Week 12

Chapters

From Transformer to LLM: the Cost Problem

1.1What you implement to build a Transformer

1.2Why full fine-tuning is out of reach

1.3Chapter summary

LoRA

2.1How many parameters?

2.2LoRA as code

2.3Chapter summary

LoRA in Practice and TensorBoard

3.1Using LoRA with the PEFT library

3.2Watching training: TensorBoard

3.3Chapter summary

Retrieval-Augmented Generation

4.1The architecture

4.2The document pipeline: load, clean, chunk

4.3Text embeddings and cosine similarity

4.4Chapter summary

FAISS, Prompts and Chains

5.1FAISS: exact and approximate search

5.2The online flow and the prompt

5.3LangChain and the "|" chain

5.4Checking a RAG system

5.5Chapter summary

The Whole Course on One Map

6.1Ideas that keep coming back

6.2What the measurements taught us

6.3A study order for the end term

Interview Question Bank

7.1Interview heat-map: what is most likely

7.2LoRA

Go deeper

Go deeper

Go deeper

Go deeper

7.3RAG

Go deeper

Go deeper

Go deeper

Go deeper

7.4Tools and choices

Go deeper

Go deeper

Go deeper

Week 12 Summary Sheet

8.1Week 12 at a glance

8.2Formulas and numbers

8.3RAG against fine-tuning

8.4Key terms

8.5The pairs you must tell apart

8.6Last-minute checklist

8.7Corrections and additions to the source notes

Deep Learning,
Week 12