Backpropagation & Autograd Engine

Backpropagation is the algorithm that computes gradients through a neural network by applying the chain rule backwards through the computation graph. Built here as a scalar-valued autograd engine — each operation tracks its inputs and knows how to propagate gradients — plus a minimal neural net library on top. ~200 lines of Python, zero dependencies.

9 notebooks: backpropagation and autograd

Notebook	Summary
01 Derivatives & Autograd	Limit definition; manual derivative computation
02 Data Struct & Forward Pass	Building the Value class; first forward pass
03 Backpropagation (by hand)	Hand-assigning gradients at each node
04 Training a Neuron	tanh as an activation function
05 Manual Backward Pass	Recursive ._backward() calls (compute each node’s gradient)
06 Automated Backward Pass	Single .backward() call, propagates via topological sort Ensure gradients accumulate (multivariate: += not =)
07 Express tanh more simply	Exercise: re-implement tanh using constituent operations
08 Pytorch: Backpropagation	Exercise: Perform backpropagation to update gradients in PyTorch
09 PyTorch: Gradient Descent	Exercise: Perform gradient descent (train the neural net) in PyTorch

Sources: Andrej Karpathy - Zero to Hero · micrograd · lectures · exercises