Onlearn

Build a WaveNet

Based on Andrej Karpathy’s course.

Python · ~120 min · tests verify your work

Start building

Use the template, open it in your editor, and tell your AI tutor to take it from there.

Use this template

Free · no signup to start · you keep the code

Just want to read the code first?

Clone it locally - no GitHub account needed. Your changes stay on your machine, so use the template above to keep your progress.

git clone https://github.com/ash25082003/onlearn-zero-to-hero-makemore-wavenet

What you’ll build

Every network you've built reads its context the same way: mash all the characters into one vector and hope a single layer sorts it out. This project changes the wiring. You promote your Module 4 classes into a mini torch.nn with real containers, stretch the context to 8 characters, and then fuse them the WaveNet way - pairs, then pairs of pairs, then pairs of those - a hierarchy that costs one .view to express. In the middle of it, the lecture's best moment happens to you: your own BatchNorm has been silently keeping four separate statistics where it should keep one, the network trains anyway, and a single printed shape gives it away. You fix the class you wrote two modules ago. The finale trains the deepest network of the course, 76,579 parameters, to the best validation number of the course in about five minutes - with a --long mode that runs the lecture's full 200,000 steps and pushes it to about 2.01.

What you’ll learn

By the end, you’ll be able to:

  • Torch-ify the stack: Embedding, Flatten, and Sequential modules with the real torch.nn API, so the model becomes one callable object
  • Scale the context window from 3 to 8 characters and articulate why a flat network spends it badly
  • Build FlattenConsecutive - the WaveNet's dilated fusion as one .view - and explain why fusing memory-adjacent neighbors is free
  • Assemble the hierarchical model and read its shape journey layer by layer: 8 characters, 4 pairs, 2 quadruples, 1 vector
  • Catch a silent bug in your own BatchNorm by reading a buffer's shape, and fix 3D normalization to pool over batch and position
  • Internalize the professional debugging habit this lecture teaches: everything can run, the loss can fall, and the model can still be wrong
  • Train the deepest network of the course to the best validation loss of the course - and know why the tree scales where the flat model stalls

Before you start: Do Module 4 first - this project starts from the very classes you wrote there (they ship in nn.py marked as yours), and one milestone is fixing a bug in one of them. Module 5 is great context but not required. PyTorch is the one install; the companion notebook is optional dessert.

Your AI tutor

It won't build the tree for you. It makes you narrate the shape journey out loud, asks how many examples a (32, 4, 68) batch really holds, and it will not spoil the bug - check.py opens the hunt, you close it.