Train a neural bigram model

What you’ll build

The same language model, learned instead of counted. Build a one-layer network, write softmax yourself, and train it with gradient descent until it converges to the loss your counting model already achieved - proof the network discovers the same knowledge. This exact framing - logits, softmax, NLL, backprop - is what scales to GPT.

What you’ll learn

By the end, you’ll be able to:

• Turn 228,146 bigrams into a supervised training set of (input, target) pairs
• One-hot encode characters and see why integers can't feed a neural net
• Write softmax by hand: logits to counts to probabilities, differentiable end to end
• Implement negative log-likelihood loss plus regularization - and see it's the counting model's smoothing in disguise
• Train with gradient descent until the network rediscovers the counting model's 2.45 loss
• Sample names from the trained network and watch it invent the same names as the counting model

Before you start: Do 'Build the bigram language model' first - this rebuilds that exact model, and the punchline lands harder if you built the original. Module 1 (micrograd) deepens the backprop intuition but isn't required; PyTorch runs the backward pass here.

Your AI tutor

It won't write the loop for you. It asks the question that makes the next line obvious, then runs the tests to check.