Train a neural net from scratch

You have an engine that computes gradients. Now put it to work. A small neural net is only a few classes on top of autograd: a neuron, a layer, a loss, and a loop that nudges every weight downhill.

A neuron is barely an equation

A neuron takes some inputs, multiplies each by a weight, adds a bias, and squashes the result through a non-linearity like tanh. In code, the old neuron metaphor becomes tanh(w · x + b). Because w, x, and b are all Value objects, this expression is automatically differentiable; the engine you built records the entire graph as it runs.

import random

class Neuron:
    def __init__(self, nin):
        self.w = [Value(random.uniform(-1, 1)) for _ in range(nin)]
        self.b = Value(random.uniform(-1, 1))

    def __call__(self, x):
        act = sum((wi*xi for wi, xi in zip(self.w, x)), self.b)  # w·x + b
        return act.tanh()

    def parameters(self):
        return self.w + [self.b]

The parameters() method matters more than it looks: it's how you'll later reach in and grab every weight to update. PyTorch's modules have the exact same method, for the exact same reason.

Layers and MLPs are simple containers

A layer is a list of neurons evaluated independently on the same input. A multi-layer perceptron (MLP) is a list of layers, where each layer's output feeds the next. No new math, just composition. The diagram you have seen of dots connected to dots is this code.

class Layer:
    def __init__(self, nin, nout):
        self.neurons = [Neuron(nin) for _ in range(nout)]

    def __call__(self, x):
        outs = [n(x) for n in self.neurons]
        return outs[0] if len(outs) == 1 else outs

    def parameters(self):
        return [p for n in self.neurons for p in n.parameters()]

class MLP:
    def __init__(self, nin, nouts):
        sizes = [nin] + nouts
        self.layers = [Layer(sizes[i], sizes[i+1]) for i in range(len(nouts))]

    def __call__(self, x):
        for layer in self.layers:
            x = layer(x)
        return x

    def parameters(self):
        return [p for layer in self.layers for p in layer.parameters()]

Now n = MLP(3, [4, 4, 1]) is a real network: three inputs, two hidden layers of four neurons, one output. Call n(x) and you get a prediction, plus a computation graph stretching from that prediction all the way back to all 41 weights.

Loss: one number for how wrong you are

To improve the network you need a single number that measures how badly it's doing right now: the loss. The simplest is mean-squared error: for each example, subtract the prediction from the target and square it (squaring discards the sign, so being wrong in either direction counts). Sum over all examples and you get one number that's 0 only when every prediction is perfect.

xs = [[2.0, 3.0, -1.0],
      [3.0, -1.0, 0.5],
      [0.5, 1.0, 1.0],
      [1.0, 1.0, -1.0]]
ys = [1.0, -1.0, -1.0, 1.0]          # desired targets

ypred = [n(x) for x in xs]
loss  = sum((yout - ygt)**2 for ygt, yout in zip(ys, ypred))

Gradient descent: nudge every weight downhill

Now use the gradients. Call loss.backward() and your engine fills in .grad for all 41 weights. Each one says how the loss responds to that weight. The gradient points in the direction that increases the loss, and we want to decrease it, so we step the opposite way. That minus sign is the single easiest thing to get wrong:

loss.backward()

for p in n.parameters():
    p.data += -0.05 * p.grad   # step AGAINST the gradient to lower the loss

The training loop

Wrap the three steps in a loop and you have training. Run it and watch the printed loss fall step by step. That number going down is your network learning.

for k in range(20):
    # forward
    ypred = [n(x) for x in xs]
    loss  = sum((yout - ygt)**2 for ygt, yout in zip(ys, ypred))

    # zero the gradients BEFORE backward (see warning below)
    for p in n.parameters():
        p.grad = 0.0

    # backward
    loss.backward()

    # update
    for p in n.parameters():
        p.data += -0.05 * p.grad

    print(k, loss.data)

So what is a neural net?

The full picture is smaller than it first seems. A neural net is a mathematical expression that takes data and weights, runs a forward pass to a prediction, and uses a loss function to score that prediction. You backpropagate the loss to get gradients, then take a small step downhill on every weight. Repeat that many times and the weights settle into values that do what you want.

Your network has 41 parameters. GPT-class models have hundreds of billions, plus a different loss (cross-entropy) and a stronger update rule (Adam, not plain SGD). But the skeleton is the same: forward, loss, backward, step. You now understand that loop at the level of individual scalars.

Now build it

Open the neural-net-training project and build it on top of the engine you already wrote: Neuron, Layer, MLP, then the training loop. The tests check that the loss drops, so you can see whether your network is learning. If you have not built the engine yet, go back to Build an autograd engine first.