STARGATE CAMPUS, ABILENE · cutaway, roofs off, illustrative layout
Stargate, Abilenethe whole campus
One Abilene buildingthe one we follow in
xAI Colossus, 2024Memphis, a whole supercomputer
ONE DATA HALL · 1 of 8 in the building · illustrative layout
POWER IN · grid to rack
Gridhigh voltage
↓
Substationsteps down
↓
Switchgearroutes, protects
↓
UPS + batteriesrides blips
↓
HEAT OUT · rack to air
CDUsloops never mix
↓
Facility water loopwarm out, cool back
↓
Chillerstandby · hottest days
↓
Dry coolers, rooffans, no water used
↓
Heat to the air
ONE DATA HALL, CUT AWAY · 3 of 16 rows drawn, network row in front, illustrative
AS A GRAPH · 8 racks
Rail-optimised: each leaf takes one GPU slot from every rack. 8-rack pod: ≈32 leaf, 12 spine.
GB200 NVL72 · side view and front view, top to bottom
SIDE VIEW
FRONT VIEW
FRONT
BACK · coolant, power
Power shelves
10 compute trays4 GPUs each
9 switch trays
8 compute trays4 GPUs each
Power shelves
←NVLink spinecopper, rack back
The tray we followcompute tray 1, opened at step 06
SCALE-UP · NVLINK · copper, every GPU to every switch chip· every GPU to every switch chip
18 NVSwitch chips
GPUs · one GPU's 18 links drawn
130 TB/s across the whole spine
POWER AND COOLING · at the back
NVLINK SWITCH TRAY · top view, back on the left
Link fanswitch chips above, GPUs below
GPU 1GPU 2GPU 3GPU 44 of the 72 GPUsSpine connectorsback edge, to the copper spine
Two NVSwitch chipsa port for each of the 72 GPUs
IN-SWITCH SUMone all-reduce, 4 of the 72 GPUs
COMPUTE TRAY · top view · back on the left, front on the right· top view
cold in
warm out
NVLink to
the spine
to other
racks
storage,
front end
WITHIN ONE RACK
BETWEEN TWO RACKS
back at the top · front at the bottom
GB200 SUPERCHIP · one board, top view
HOW MUCHmemory one GPU can reach
HBM3eown package
186 GB
Grace LPDDR5Xover C2C
480 GBshared by 2
HOW FASTread speed from one GPU
HBM3e
8,000 GB/s
Grace LPDDR5X
450 GB/s
x86 serverCPU RAM over PCIe 5
64 GB/s
One GB200 GPU reading; per direction where it matters.
B200 GPU PACKAGE · top view, drawn with its real counts
HBM3e24 GB
HBM3e24 GB
HBM3e24 GB
HBM3e24 GB
HBM3e24 GB
HBM3e24 GB
HBM3e24 GB
HBM3e24 GB
GB100 die
GB100 die
die-to-die link · 10 TB/s
SUBSTRATE · INTERPOSER
NVLINK OUTto the other 71 GPUs
From this GPU: 18 links900 GB/s each way
OTHER RACKS PCIe → ConnectX-7 → leaf switch
ONE SM · four partitions around shared memory
ONE TILE MULTIPLY · C = A × B, 4 × 4, illustrative values
One 4,096 × 4,096 weight matrix in Llama 3.1 8B, cut into 256 tiles of 256 × 256 · tile size illustrative
the tile drawn above, shrunk to 4 × 4
HBM3E STACK · side section, not to scale vertically
DRAM dies
3 GB each
TSVs
copper, vertical
base die
talks to the GPU
interposer
the bus, sideways to the die
HBM3eone stack
1,024 wiresDDR5one DIMM
64 wiresInterface width only, not speed.
B200 AND B300 · same package, per-GPU figures
ONE STACK, SIDE VIEW
B300: 12 high
B200: 8 high
base die
LINEAGE· HBM per GPU, by year of first shipment
H1002022
80 GB
HBM3H2002024
141 GB
HBM3eB2002024
186 GB
HBM3e · 8-highB3002025
278 GB
HBM3e · 12-highRubin2026
288 GB
HBM4 · announcedONE MODEL, SIX FORMATS · Llama 3.1 8B, 8.03 B weights · bits drawn to scale
FP3232 bits · E8M23
BF1616 bits · E8M7
FP88 bits · E4M3
FP44 bits · E2M1
Q8_08.5 bits avg
Q4_K_M4.9 bits avg
EVERY POSITIVE FP4 VALUE · before the scale
TOKENIZER · text in on the left, ids out on the right
HOW OTHER TEXT GETS CUT· the same tokenizer, each piece with its id
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
EMBEDDING LOOKUP · real shapes · matrix not to scale, broken below row 10,000
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
RESIDUAL STREAM · isometric · blocks run left to right, tokens go into the page
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
ONE BLOCK · block 1 of the stream in 03, zoomed in · attention first, then the MLP
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
QKV · one token, three projections, each cut into heads · hover: head 1
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
SCORES · how much each token listens to each earlier one · one head
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
HEADS · 06 showed head 1 · there are 32, each with its own slice of q and k · ·bank’s row outlined
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
BLENDING VALUES · the row for ·bank, 3 of 128 numbers shown, illustrative
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
MLP · SwiGLU · one token, left to right · 4,096 = 52 px, to scale · hover a W for its maths
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
THE STACK · the residual stream runs left to right
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
OUTPUT HEAD · the last vector scored against every token, illustrative logits· illustrative logits
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
FOUR PASSES · each row is one run of the whole model
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
KV CACHE · one column per token, 32 layers each
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
FOUR GROUPS · by what they fix
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
BYTE-PAIR MERGES · "·opened", illustrative merge order
Merge order and pair counts illustrative. · is the leading space.
TOP PAIRS · before merge 1
05 M9.4 M
TIMES SEEN, MILLIONSThe winner becomes merge 1. Then every count is redone.
BYTE PATCHES · the same prompt, patch edges illustrative
Entropy from a small byte model; values illustrative.
ENGRAM · hashed n-gram lookup beside the stream, illustrative rows
27B models, same parameters and FLOPs; gains from the paper's Table 1, NIAH from Table 2. Rows and hash count illustrative.
HYPER-CONNECTIONS · one token, four streams, Llama width for scale
HC · each mixing matrix free to take any values
ATTENTION OVER DEPTH · one token at one layer, weights illustrative
RMSNORM · the ·bank vector before and after, first 4 of 4,096 numbers
INPUT → OUTPUT · one number, curves illustrative
KV CACHE PER TOKEN · Llama 3.1 8B shape, BF16
SPARSE ATTENTION · one query over a long context, illustrative scores
EXCLUSIVE SELF-ATTENTION · "·bank", first 3 of 128 numbers, illustrative values
ACTIVATIONS · output against input, one hidden unit
MIXTURE OF EXPERTS · DeepSeek-V3 layer, one token
EXPERT LOAD · share of tokens per expert, illustrative
LATENT MoE · Kimi K3, one token through one layer
STABLE LATENT MoE · three guards, Kimi K3
LOGIT MARGIN · target logit minus log-sum-exp of the others
MEMORY PER CHAT · a Llama-shaped model, if its attention were linear
STATE UPDATE · one head, illustrative values
CSA AND HCA · one million tokens of context, DeepSeek-V4-Pro
KV CACHE AT ONE MILLION TOKENS · relative to a grouped-query baseline
SPECULATIVE DECODING · one big-model pass, illustrative draft
ACCEPT RULE · big model p against draft q, one position
TOKENIZER · text in on the left, ids out on the right
HOW OTHER TEXT GETS CUT· the same tokenizer, each piece with its id
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
EMBEDDING LOOKUP · real shapes · matrix not to scale, broken below row 10,000
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
RESIDUAL STREAM · isometric · blocks run left to right, tokens go into the page
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
ONE BLOCK · block 1 of the stream in 03, zoomed in · attention first, then the MLP
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
QKV · one token, three projections, each cut into heads · hover: head 1
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
SCORES · how much each token listens to each earlier one · one head
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
HEADS · 06 showed head 1 · there are 96, each with its own slice of q and k · ·bank’s row outlined
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
BLENDING VALUES · the row for ·bank, 3 of 128 numbers shown, illustrative
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
MLP · GELU · one token, left to right · 12,288 = 52 px, to scale · hover a W for its maths
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
THE STACK · the residual stream runs left to right
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
OUTPUT HEAD · the last vector scored against every token, illustrative logits· illustrative logits
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
FOUR PASSES · each row is one run of the whole model
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
KV CACHE · one column per token, 96 layers each, 3 per stripe
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
FOUR GROUPS · by what they fix
INSIDE A TRANSFORMER · Vaswani et al. 2017, Fig. 1 and 2
BYTE-PAIR MERGES · "·opened", illustrative merge order
Merge order and pair counts illustrative. · is the leading space.
TOP PAIRS · before merge 1
05 M9.4 M
TIMES SEEN, MILLIONSThe winner becomes merge 1. Then every count is redone.
BYTE PATCHES · the same prompt, patch edges illustrative
Entropy from a small byte model; values illustrative.
HYPER-CONNECTIONS · one token, four streams, GPT-3 width for scale
ATTENTION OVER DEPTH · one token at one layer, weights illustrative
LAYERNORM · the ·bank vector before and after, first 4 of 12,288 numbers, illustrative
KV CACHE PER TOKEN · GPT-3 175B shape, BF16
ACTIVATIONS · output against input, one hidden unit
MIXTURE OF EXPERTS · DeepSeek-V3 layer, one token
LOGIT MARGIN · target logit minus log-sum-exp of the others
MEMORY PER CHAT · a GPT-3-shaped model, if its attention were linear
SPECULATIVE DECODING · one big-model pass, illustrative draft
ACCEPT RULE · big model p against draft q, one position
Hover a part to highlight it. Click it to read about it.