A visual walkthrough of micrograd — Andrej Karpathy's ~150-line autograd engine. No matrices, no vectors. Just scalars, a computation graph, and the chain rule you already know. By the end you'll have watched a real backward pass happen, one node at a time.
Forget neural networks for a moment. We have two inputs,
a = 3 and b = 4, and we build up a result through a few simple steps.
At each step we remember which numbers produced it and which operation was used —
that memory is what turns arithmetic into a "computation graph".
Read it left to right: inputs flow into operations, operations produce new values,
those values feed the next operations, until we reach the final output e = 84.
Notice something important about a —
it's used twice: once to make c, once to make d. Hold onto that thought.
The forward pass is the easy half. The interesting question — the one backpropagation answers —
is the reverse: "if a wiggled a little, how much would e move?"
In calculus class you'd call that de/da. In code, we call it a.grad.
Everything in micrograd is a Value. It's just a number wrapped together with four extra fields
that record how it came to exist. That's the whole trick: by remembering its own birth,
a number becomes a node in a graph that knows how to differentiate itself.
Five fields. Don't gloss over them — the entire engine is just these five fields being read and written.
| Field | Holds | Plain meaning |
|---|---|---|
| data | a float | The number this node represents. |
| grad | a float, starts 0 | How much the final output changes if this node changes. Filled in during backward. |
| _prev | set of Values | My parents — the inputs that produced me. Empty for leaves like a and b. |
| _op | a string | The symbol of the operation that made me ('+', '*', 'ReLU'…). |
| _backward | a function | A closure that pushes my received gradient back to my parents. The heart of the engine. |
A Value never "looks forward". It only knows its parents (in _prev), and it carries a
_backward recipe for handing gradients back to those parents. Reverse-mode autodiff is just
a wave of these recipes firing in the right order.
Here is the most important mental shift in the whole subject.
When c = a + b runs, the code does two things, not one:
c.data = a.data + b.data and remembers c._prev = {a, b}._backward and attaches it to c, to be run later.
That closure doesn't run yet. It's a sticky note left on c saying:
"when the time comes, here's how I forward my gradient to my parents."
The local derivative of a sum with respect to either input is 1 — so each parent
receives exactly what out received, undiminished. That's the whole + operator.
For out = self · other, the derivative w.r.t. self is other (and vice versa).
So the parent receives other.data * out.grad — that multiplication
is the chain rule, written out: local gradient × upstream gradient.
And that's the whole toolkit. Subtraction is a + (-b); division is a · b⁻¹;
sin, tanh, exp slot in the same way — each just needs its own one-line
derivative in a closure. micrograd ships only +, *, **, and relu
because everything else composes from those four, no new machinery required.
+= and not =?
Because a node can reach the output through multiple paths.
Remember a from §1? It feeds both c and d,
and both eventually reach e. So a receives a contribution from each path,
and the total derivative is the sum.
In proper calculus this is the multivariable chain rule
(the total-derivative theorem). If a value x reaches the output L through
intermediate nodes y1, y2, …, yk,
then:
That giant summation symbol Σ on the right — the thing that says "add up all the paths" —
that is the += in the code. They are the same operation. If you wrote = instead,
each new path would overwrite the previous contribution and the gradient would silently come out wrong.
The × in the chain rule is the other.data * out.grad multiplication.
The + in the chain rule is the +=. They are different jobs, done by different pieces of the code.
Here is the same graph e = (a+b)·(a·b), rendered the way micrograd's own visualizer draws it:
each rectangle is a Value split into data (left) and grad (right).
Press Step to walk backward through the graph in topological order.
Watch the gradients fill in — and pay special attention to a:
it lights up twice, once per path.
Look back at what just happened. The gradient reached a only after
both c and d had finished. That's not a coincidence — it's a hard requirement:
A node may run its _backward() only after every node that uses it
has already run its own. Otherwise node.grad wouldn't yet contain all its contributions.
micrograd enforces this with a topological sort: a linear ordering of the nodes where every node appears after all of its inputs. Build it with a depth-first walk from the output:
Two subtleties worth fixing in your head. First, build_topo must start at the
output — leaves like a have empty _prev, so starting from one
would only ever visit that single node. Second, the resulting topo runs
leaves-first; the engine reverses it so recipes fire output-first, which is the order
you just stepped through above.
That's it. That's the entire backward() method: build a topo order, set
output.grad = 1, walk it backwards, and let each node's pre-staged closure do its one tiny job.
A single Value differentiates itself. A neuron is just a small graph of Values —
act = Σ(w·x) + b passed through an activation. A layer is a list of neurons.
A network is a list of layers. The composition is so clean it almost reads like a definition:
Because every weight w and bias b is itself a Value, the moment you
compute a loss the entire network is automatically a giant computation graph — and the same
backward() you just watched fills in every parameter's gradient. Training is then a four-line loop:
The same six-field object and the same topo-sorted wave of closures that handled a + b
also handle a 50-layer ResNet. The mechanics don't change — only the size of the graph does.
micrograd operates on scalars, so it's slow — you'd never train a real model with it. But every idea
above — the stored local recipe, the += over paths, the topological wave — is exactly what
PyTorch and JAX do internally. They just batch the work into big matrix operations and hand it to a GPU.
Once you've seen it here, at the scale of five numbers, you have the whole picture.