Hardtransformersattentiongeneration8 steps
Build a GPT from Scratch
Tokenize text, embed it, run causal self-attention, normalize, and generate the next token — a decoder-only transformer assembled problem by problem.
Your progress
0/8Solve a step and it is marked here automatically — progress stays in this browser.
Steps
Work through them in order — each step builds on the last.- Not solvedNextEasy
- Not solvedToken Embedding LookupNextEasyDeep Learning
- Not solvedSinusoidal Positional EncodingNextMediumDeep Learning
- Not solvedAdd Positional EncodingNextEasyDeep Learning
- Not solvedCausal Attention ScoresNextHardDeep Learning
- Not solvedRow-Wise SoftmaxNextMediumDeep Learning
- Not solvedLayer NormalizationNextMediumDeep Learning
- Not solvedGreedy Token GenerationNextEasyDeep Learning