Build the bigram language model
Based on Andrej Karpathy’s course.
Python · ~60 min · tests verify your work
Start building
Use the template, open it in your editor, and tell your AI tutor to take it from there.
Use this templateFree · no signup to start · you keep the code
Just want to read the code first?
Clone it locally - no GitHub account needed. Your changes stay on your machine, so use the template above to keep your progress.
What you’ll build
A working language model - the same idea as GPT, shrunk to its simplest possible form. It reads 32,033 real names, learns which letter follows which, and invents new ones: ka, teda, moliellavo. You build every piece: the counts, the probabilities, the sampler, the loss.
What you’ll learn
By the end, you’ll be able to:
- Load a real dataset of 32,033 names and build a character vocabulary with a start/end token
- Count all 228,146 bigrams into a 27x27 tensor - your first language model is a table
- Normalize counts into probabilities and survive the keepdim broadcasting trap on purpose
- Sample new names from the model with torch.multinomial
- Score the model with average negative log-likelihood and learn why 2.454 is the number to beat
Before you start: Comfortable with Python loops and dicts. No calculus and no neural networks - this is the gentlest project in the course. PyTorch is the one install, and it's taught from zero.
Your AI tutor
It won't hand you the answer. It asks the one question that makes the next line obvious, then checks your work by running the tests.