Backpropagation is the algorithm that computes gradients through a neural network by applying the chain rule backwards through the computation graph. Built here as a scalar-valued autograd engine — each operation tracks its inputs and knows how to propagate gradients — plus a minimal neural net library on top. ~200 lines of Python, zero dependencies.
9 notebooks: backpropagation and autograd
01
Derivatives & Autograd
Limit definition; manual derivative computation
02
Data Struct & Forward Pass
Building the
Value class; first forward pass
03
Backpropagation (by hand)
Hand-assigning gradients at each node
04
Training a Neuron
tanh as an activation function
05
Manual Backward Pass
Recursive
._backward() calls (compute each node's gradient)
06
Automated Backward Pass
Single
.backward() call, propagates via topological sort. Ensure gradients accumulate (multivariate: += not =)
07
Express tanh more simply
exercise
Re-implement
tanh using constituent operations
08
Pytorch: Backpropagation
exercise
Perform backpropagation to update gradients in PyTorch
09
PyTorch: Gradient Descent
exercise
Perform gradient descent (train the neural net) in PyTorch
Sources: Andrej Karpathy - Zero to Hero · micrograd · lectures · exercises