Tokenization: Byte-Pair Encoding

A language model does not read text. It reads a sequence of whole numbers, and a tokenizer is the program that turns text into those numbers. That choice decides the length of every prompt, the price of every call, and how the model handles unusual words.

The simplest tokenizer splits on spaces. It breaks on don't, on state-of-the-art, and on every language that does not put spaces between words. It also throws away capitalization and punctuation unless you write a special rule for each case.

Splitting into single characters avoids all of that. Nothing is ever unknown, but a sentence becomes long, and long sequences cost more to train and to serve.

Treating every whole word as one token fails the other way. A vocabulary holding every form of every word in a language that builds many forms from one root is enormous, and any word outside it has no number at all. A new name or a typo becomes a blank.

Subword tokenization is the middle road. Frequent pieces stay whole; rare words break into reusable parts. The model sees low inside lowest, and est remains a shared ending.

Byte-pair encoding, BPE for short, builds those pieces from the data. It starts with a small alphabet: every character in the corpus — the training text — plus a marker for the end of a word, written </w>.

Then it counts every neighboring pair of symbols across the corpus and merges the most frequent pair into one new symbol. That merge goes onto an ordered list, and the counts are refreshed. Repeat.

Every round performs the same move. What comes out is a ranked list of merges. That list plus the starting alphabet is the entire tokenizer.

What you just stepped through is greedy: each round takes the single most frequent pair. That is not the theoretically best vocabulary, but it is fast, repeatable, and still standard in 2026.

Two details matter. Ties are broken by the order of the scan, so scan direction is part of the algorithm. And merges apply left to right, so one merge can hide a pair that another merge would have used.

After training, encoding is one pass: split the text, mark the end of every word, then apply the merges in rank order. How many tokens come out is the cost. It sets the context budget, the response time, and the price of an API call.

Fertility is the average number of tokens per word. English runs about 1.3. A language whose writing system the tokenizer rarely saw during training can run 5 or 10. The same sentence then costs several times more — tokenizer quality is a fairness problem, not just an engineering one.

Byte-level BPE skips the character alphabet and starts from the 256 possible byte values instead. Every string is representable that way, so there is no unknown token and no crash on emoji, unusual names, or code.

The price moves into length. Byte-level tokenizers need more tokens for the same text, so the effective training sequence grows. Most modern models accept that trade.

Special tokens sit on top of the learned vocabulary. Markers such as [BOS] (beginning of sequence), [EOS] (end), and [PAD] (filler), plus chat roles and tool markers, receive reserved numbers the merge process never touches.

A tokenizer is frozen when training starts, because the model's embedding table is indexed by token number. Swap the tokenizer later and every learned vector points at the wrong symbol.

The problems below start with counting tokens, move through byte-level BPE and WordPiece (another subword method), and end at fertility and characters per token.