Neural Networks: Zero to Hero

A hands-on course that builds neural networks from scratch, one video at a time, from a tiny automatic-differentiation engine up to a GPT. Each module is one lecture; each module has one or more guided coding projects you complete locally in Claude Code with an AI tutor.

Start the track

Work through it locally with your AI tutor. It checks each project against the tests.

Start Micrograd

Free · no signup to start · you keep the code

Companion guides

Roadmap

Build from micrograd to GPT.

LiveLecture 1

The spelled-out intro to neural networks and backpropagation: building micrograd

Open

Project 1 · python

Build the autograd engine

Project 2 · python

Build & train a neural net

LiveLecture 2

The spelled-out intro to language modeling: building makemore

Open

Project 1 · python

Build the bigram language model

Project 2 · python

Train a neural bigram model

LiveLecture 3

Building makemore Part 2: MLP

Open

Project 1 · python

Build an MLP language model

LiveLecture 4

Building makemore Part 3: Activations & Gradients, BatchNorm

Open

Project 1 · python

Fix your init & build BatchNorm

LiveLecture 5

Building makemore Part 4: Becoming a Backprop Ninja

Open

Project 1 · python

Become a backprop ninja

LiveLecture 6

Building makemore Part 5: Building a WaveNet

Open

Project 1 · python

Build a WaveNet

LiveLecture 7

Let's build GPT: from scratch, in code, spelled out.

Open

Project 1 · python

Build a GPT

LiveLecture 8

Let's build the GPT Tokenizer

Open

Project 1 · python

Build the GPT Tokenizer

LiveLecture 9

Let's reproduce GPT-2 (124M)

Open

Project 1 · python

Reproduce GPT-2

What you’ll learn

Automatic differentiationComputation graphBackpropagationChain ruleReverse-mode autodiffGradientsNeural networksMulti-layer perceptronGradient descentLoss functionsForward and backward passTraining loopLanguage modelingBigramsProbability distributionsPyTorch tensorsBroadcastingSamplingNegative log-likelihoodSmoothingOne-hot encodingLogitsSoftmaxRegularizationPyTorch autogradEmbeddingsHidden layersMLPsCross-entropy lossMinibatch SGDTrain/dev/test splitsOverfittingLearning-rate tuningTensor internalsInitializationTanh saturationDead neuronsVanishing gradientsKaiming initBatchNormRunning statisticsTrain/eval modesTraining diagnosticsThe chain rule at tensor scaleMatmul gradientsBroadcasting in reverseGradient accumulationSoftmax/cross-entropy backwardBatchNorm backwardScatter-addNumerical exactnessManual optimizationModules and containersHierarchical fusionview/reshape semanticsBatchNorm over 3D inputsBroadcasting-hidden bugsShape-driven debuggingContext scalingArchitecture as a scaling betEqual-budget comparisonsNext-token predictionSelf-attentionQueries, keys, and valuesCausal maskingMulti-head attentionResidual connectionsLayerNormPositional embeddingsThe Transformer blockAutoregressive generationUTF-8 and code pointsByte-pair encodingPair statisticsThe merge operationVocabulary mintingMint-order encodingRegex pre-splittingSpecial tokensCompression ratioWhy LLMs can't spellFused multi-head attentionWeight tyingGELU and frozen approximationsResidual-stream init scalingState-dict surgeryCheckpoint fingerprintingTemperature and top-k samplingWarmup-cosine schedulesLM-based eval scoring