Build a Transformer from Scratch

Everything starts as text: byte-level BPE token counts and merge steps, embedding lookups and parameter budgets, the backward scatter-add that trains them, and sinusoidal positional encoding.

The core mechanism: scaled attention scores, softmax weights, and the weighted sum, then multi-head split and merge, a parameter count with bias, and cross-attention shapes.

Wrap attention in the pieces that make it trainable: LayerNorm and its RMSNorm comparison, GELU in both tanh and exact forms, pre- versus post-norm residual scaling, and a full encoder block with its parameter count.

Teach the stack to predict: numerically stable softmax, the softmax cross-entropy backward pass, padding and sliding-window attention masks, seeded attention dropout, and a mixed-precision loss-scaling step.

Run it and count the cost: KV-cache sizing and appends, online-softmax rescaling and logsumexp tracking, quadratic attention FLOPs, and forward FLOPs per token.