A big cloud data centre · ≈30 MW

A frontier AI campus · ≈1 GW

Gold: the hall on the next screen. Each one draws ≈17 MW, cooling included.

Plans now run to several gigawatts. Sites are picked for power first: a grid connection, or gas for turbines on site. Land is the easy part.

A data centre for AI is described by how much power it draws, not how big it is. Every number on the next screens comes out of this budget.

Every watt that goes in at the top comes out as heat on the right. The racks in between turn a little of it into tokens.

NVLink, inside the rack

This network, between racks

18–36× slower. Work that talks constantly stays inside a rack; lighter traffic crosses here.

Any GPU in the hall can reach any other in a few hops, at full speed for every GPU at once. It's still far slower than the copper inside a rack, so jobs are laid out around that gap.

18 NVSwitch chips · 2 per switch tray

72 B200 GPUs · each column is one compute tray

One GPU's 18 links drawn in full; every GPU has the same. More than 5,000 copper cables run down the back of the rack.

This rack (gold) and thousands more · 400 Gb/s per GPU

Blue in, brick out: heat leaves in water

Cold plates sit on all 6 chips; quick-disconnects at the back let a tray slide out without draining the rack.

1.8 TB/s per GPU, 18–36× the cluster network: work that needs constant communication stays inside the rack. Each GPU sends 1/18 of its traffic through each of the 18 chips.

How much

How fast

Grace's memory is a slower overflow tier: ≈9× slower than HBM, but ≈7× faster than reaching a CPU over PCIe, and the GPU reads it directly.

In an ordinary server the CPU sits behind PCIe. Here it's joined to its GPUs by a fast, coherent link, so its memory works as overflow for theirs.

8 of the 71 other B200s

Every GPU, this one too, has 18 links: one to each switch chip. Any GPU reaches any of the other 71 in one hop.

4 schedulers

4 × 64 KB

128 CUDA cores

4 tensor cores

A · 16 × 16

×

B · 16 × 8

+

C · 16 × 8

=

D · 16 × 8

BF16

FP8 (E4M3)

FP4 (E2M1)

Halve the bits and the same silicon does twice the multiplies, and every weight takes half the memory and bandwidth. That's why the move to FP8 and FP4 matters so much.

TSVs: copper wires drilled through every die, straight down to the base.

The stack sits millimetres from the compute die, joined through the interposer.

Why stack? A memory stick has a 64-bit bus. Stacking puts 1,024 wires within millimetres of the GPU, which is where the bandwidth comes from.

Decode reads every weight once per token, so a B200 can stream its whole 192 GB about 40 times a second at most.

B200

B300

HBM3e

longer contexts

Dense FP4

+50%, vendor figures

Attention exp()

faster softmax in the SFUs

NVLink

unchanged

Scale-out NIC

ConnectX-8

Power

per GPU in an NVL72

FP64

Ultra drops HPC maths

1 · Text → tokens

2 · Token ids → vectors

Each id picks a row from a 128,256 × 4,096 table: 4,096 numbers per token (8 drawn).

3 · 32 transformer blocks

4 · Last vector → scores

One score (logit) for every token in the vocabulary.

5 · Scores → probabilities

6 · Pick one

7 · Append and repeat

“The capital of France is Paris” goes back in. Only the new token runs through the blocks; the others are remembered in the KV cache.

Common words: one token

Rare words: several pieces

Numbers: up to 3 digits each

Same word, different ids

English averages about 4 characters a token. Code and other languages usually take more tokens for the same text.

The model never sees text, only these ids. Everything it learns about spelling, it learns from which ids tend to follow which.

Rows start random. Training moves them so tokens used in similar ways end up with similar rows.

It's only a starting point: the same row for “ capital” whether it means a city or money. The blocks after this work out which.

Each vector is the token's starting point in the residual stream. Everything the model does from here is added onto it.

One number of “is”, through the first blocks

Every layer reads the stream and adds something back. Nothing is overwritten, so early information survives to the end, and a layer that has nothing to add can add almost zero.

Next, “is” compares its query with every key, then takes a mix of the values.

Every 4 query heads share one key and one value head, so the KV cache is 4× smaller.

Attention is the only step where tokens share information. Everything else in the model works on one token at a time.

Dashed cells are later tokens, masked out before softmax.

1 · Score: q(is) · k / √128

2 · Softmax: exponentiate, divide by the total

Scores 0.5 apart become weights several times apart: the biggest score dominates.

The causal mask is why a chat model can only look backwards: when it writes a token, the tokens after it don't exist yet. Weights here are illustrative, not from a real model.

52% of France's value and 32% of capital's: “is” now carries “capital of France”.

Gold: head 7, the one on the left. Each head gathers something different; W_O combines them all.

After attention, “is” carries the context it needs: the next layers can tell that what follows should be a capital, and France's.

Attention moves information between tokens: “is” learns it's about France.

The MLP looks things up: France + capital → Paris. Most of the facts live here.

Each step adds a small change rather than replacing the vector. That's what lets 32 blocks stack.

Rows sum to 1. Dashed cells are masked: a token can't see the future.

Real heads specialise: some track the previous word, some match brackets, some look up the subject. The weights in this grid are illustrative, not from a real model.

Each hidden unit tests the token for one pattern and, if it matches, writes a little back. Every unit runs, every token: 176 M multiply-adds per layer.

Total parameters grow with the number of experts; compute per token grows only with the number used. Models like DeepSeek-V3 (671 B total, 37 B active) and Qwen3 MoE work this way.

MoE trades memory for speed: all 671 B parameters must sit in GPU memory, but each token only pays for 37 B. That's why big MoE models want racks like the NVL72.

A 4090 is much faster than a 3090 at prefill, but decode barely moves: its bandwidth is only ≈8% higher. For local models, memory and bandwidth are the specs that matter.

×

=

One weight, 2,000 uses: the tensor cores, not memory, set the pace.

prefill · 2k tokens

decode · 1 chat

Prefill sits on the flat roof: compute-bound.

Every token leaves a key and value in every layer: 2,000 × 128 KB ≈ 262 MB. Decode reads it back on every step.

Prefill is the reading half: it sets time to first token, and it's the half that loves raw FLOPs (and lower precision like FP8).

Each token reads every weight once, so halve the bytes and you double the ceiling. Real speeds land below it.

Dark blue = prefill, green = decode, dashed = idle. The GPU waits on you most of the time, and when it works it's batch 1: far left on the roofline.

Laptops and phones run the same maths with less memory and bandwidth, so they use smaller models (≈1–8B) at 4-bit.

Router picks 8 of 256 experts

Dispatch: the token's vector goes to those 8 GPUs (all-to-all)

Each expert runs its small MLP

Combine: 8 answers come back, weighted, and are summed

Twice over NVLink per MoE layer, ≈60 layers, every step.

The rack acts like one giant GPU: 72 GPUs share one NVLink fabric, so experts can live anywhere in it.

“of France” is a phrase; “is” needs a subject.

“capital of France” becomes one idea, carried on “is”.

capital of France → Paris. The last vector now points at “ Paris”.

A tendency found by interpretability research, not a rule: the jobs overlap and vary by model.

Stacking the same block is the whole model. Depth is what lets it build from words to meaning to the answer, one small addition to the stream at a time.

The model never outputs a word. It outputs a probability for every token in its vocabulary, and the sampler makes the choice: at temperature 0, always “ Paris”.

Time, one chat

Each pass needs the token the last one picked, so they can't overlap. Speed is counted in tokens per second.

The prompt's 5 tokens took one pass together. Every new token costs a pass of its own.

Generation is a loop: run the model, pick a token, append it, run again. That's why a model's output speed is measured in tokens per second.

One chat's cache, by length

At 128K tokens one chat's cache is as big as the whole model. Without it, every pass would redo every earlier token's K and V in all 32 layers.

Each new token computes only its own keys and values and reads everyone else's from the cache. It trades memory for compute, and it grows with every token.

Generating a token reads every weight and does about one multiply per byte, so the GPU waits on memory. Almost every trick here reads fewer bytes, or gets more tokens from each read.