A bigram model predicts the next character based only on the current one — the simplest possible language model. Built first as a counting model (character pair frequencies → probabilities), then rebuilt as an equivalent single-layer neural network trained with gradient descent, showing both approaches converge to the same solution.
5 notebooks + 2 helpers: bigram model and NN equivalent
01
Define Bigram Model
Character pair counting; building the probability table
02
Sampling
Generating names by sampling from the distribution
—
Broadcasting Tensors
helper
PyTorch broadcasting rules with worked examples
03
Loss & Smoothing
Negative log-likelihood loss; Laplace smoothing
04
Bigrams -> Neural Net
One-hot input -> linear layer -> softmax; mathematically equivalent to counting
—
One-Hot Encoding
helper
How one-hot vectors encode categorical inputs
05
Optimisation
Training loop; gradient descent convergence
Sources: Andrej Karpathy - Zero to Hero