A transformer you can take apart
by David Zhang
introduction.mp4. In dev the videos/
folder is served automatically; for a deployed site set VITE_VIDEO_BASE.
Hover any box to open its explanation — the boxes marked ▶ have a video. Click to keep the panel open while you watch. Drag the ◇ grips or use + / − to add and remove heads and blocks, and click a ✚ to delete a residual connection.
Tap any box for its explanation — the boxes marked ▶ have a video. Use + / − to add and remove heads and blocks, and tap a ✚ to delete a residual connection. Pinch or use the zoom buttons to read the labels.
nanogpt.py — the file every explanation on this page quotes, start to finish.