Build an MLP language model

What you’ll build

The bigram hit a wall: one character of context, loss stuck at 2.454, and no cheap way to widen a counting table. This project breaks through it with the model from Bengio et al. 2003 - embeddings that give every character learned coordinates, a tanh hidden layer, minibatch training, and an honest train/dev/test split. The finale is a number: your dev loss beats 2.454. And the architecture you built - embed, hidden, logits, cross-entropy - is GPT's skeleton. A companion notebook draws the payoff: the learning-rate sweep, and the embedding map where the vowels cluster on their own.

What you’ll learn

By the end, you’ll be able to:

• Slide a 3-character context window over 32,033 names - same 228,146 examples as the bigram, three times the history
• Embed characters with a lookup table and see why C[X] plus .view() is the whole trick
• Build a real hidden layer - tanh(emb @ W1 + b1) - and a forward pass that stops at logits
• Swap your hand-written softmax loss for F.cross_entropy and know exactly why it's safer
• Train on minibatches and read a noisy loss curve without panicking
• Split train/dev/test and evaluate honestly - loss on held-out data is the only number that counts
• Beat the bigram: push the dev loss under 2.3 and sample names that finally sound like names - makilah, kaley, dakstin

Before you start: Do Module 2 (the bigram projects) first - this project exists to beat that model, and it reuses machinery you built there: the '.' token, average NLL, multinomial sampling. PyTorch is the one install; a companion Jupyter notebook is included but optional.

Your AI tutor

It won't build the network for you. It asks the question that makes the next line obvious, then runs the tests to check - and it knows where this lecture bites: the .view() trick, the noisy loss curve, the loss that starts at 26.