Build a GPT

What you’ll build

This is the one. You write a GPT - the same decoder-only Transformer architecture as GPT-2 and GPT-3, smaller numbers - from empty files, and train it on the complete works of Shakespeare until it speaks. Attention gets built from its central trick outward: one triangular masked-softmax matrix, then a head with queries, keys, and values, then many heads, then the block with residuals and LayerNorm - each step verified by tests, including the one that matters most: the future must never leak into the past. And after six modules of building everything by hand, this is where it pays off as imports - nn.Linear is your Module 4 class, nn.Embedding is Module 6's, LayerNorm answers your BatchNorm questions, AdamW retires your hand-rolled SGD. The finale runs in three acts: the bigram posts its bar, your GPT beats it by half a nat in about five minutes on a laptop CPU, and then the model speaks - 500 characters grown from a single newline, speaker names and all.

What you’ll learn

By the end, you’ll be able to:

• Build the language-modeling data pipeline: character tokenizer, 90/10 split, and the batcher where every chunk packs block_size prediction problems
• Rebuild the bigram model as a proper nn.Module with a generate() loop - and post the baseline your GPT must beat
• Derive the trick at the heart of attention: causal averaging as one masked-softmax matrix multiply
• Build a real attention head - queries, keys, values, the 1/sqrt(head_size) scaling, and the mask that makes a decoder a decoder
• Compose multi-head attention and the per-token feed-forward layer, and prove with tests that communication and computation stay separate
• Assemble Transformer blocks with residual connections and pre-norm LayerNorm - and know why that recipe lets GPTs go deep
• Put it all together - token embeddings, position embeddings, the stack, the head - and train it with AdamW until it writes Shakespeare-shaped verse

Before you start: The course so far, especially Modules 2, 4, and 6 - the sampling loop, the init and train/eval lessons, and the container fluency all return here. This is the module where torch.nn opens up, and it lands hardest if you built the hand-rolled versions first. PyTorch is the one install; the companion notebook is optional dessert.

Your AI tutor

It won't write the Transformer for you. It asks what row t of the attention matrix means, makes you explain the mask before you type it, and treats a failing causality test as the best teaching moment in the course.