A transformer you can take apart

GPT Explorer

by David Zhang

GPT Explorer

by David Zhang · build a transformer by dragging it, train it on your own hardware, read the code behind every piece
engine: starting…

Hyperparameters

16
rows trained in parallel · live
32
context length · sizes the position table (rebuild)
64
embedding width, the “C” that flows through (rebuild)
4
independent attention heads (rebuild)
16
per-head key/query/value width (rebuild)
2
stacked transformer blocks (rebuild)
0.00
applied only while training · live
3.0e-3
Adam · applied live
vocab_size (from data)65
parameters

Shape check

All matrix shapes
⚠ structure changed — Rebuild to apply it to the model (resets weights).

Engine

Training runs entirely on your machine.

Hover any box to open its explanation — the boxes marked have a video. Click to keep the panel open while you watch. Drag the grips or use + / − to add and remove heads and blocks, and click a to delete a residual connection.

Tap any box for its explanation — the boxes marked have a video. Use + / − to add and remove heads and blocks, and tap a to delete a residual connection. Pinch or use the zoom buttons to read the labels.

100%
i/o tensor x (…, C) block / attention compute logits / loss

Try my trained model

checking for the reference checkpoint…
— press the button —

Train your own

step0
loss
ms / step
steps / sec

Loss

Sample

— train a bit, then Generate —

Training data

characters
vocabulary
Preview / paste your own

The full model code

nanogpt.py — the file every explanation on this page quotes, start to finish.

Download