Byte-pair encoding: watch a model chop text into tokens

This week the machine learns to read. Not the meaning, that comes later, but the raw first step: chopping your text into tokens. Every chatbot does this before anything else, and it explains a lot of their quirks, like why they miscount letters and why some words cost more than others.

The gadget below is a byte-pair encoding (BPE) tokenizer, the scheme behind GPT-style models. Type some text and drag the Merges slider (or press Animate).

Watch what happens:

1. At 0 merges, every character is its own token. “the” is three tokens: t, h, e. The token count equals the character count.
2. Each merge finds the most frequent adjacent pair in your text and fuses it into one new token. First maybe “t” and “h” become “th”, then “th” and “e” become “the”.
3. As merges pile up, common letters grow into subwords and whole words, and the token count drops. The list of merges is the beginning of a vocabulary.

Why models read pieces, not letters or words

A model needs a fixed vocabulary of pieces. Letters keep the vocabulary tiny but make the sequences painfully long; whole words make sequences short but the vocabulary explodes and every typo becomes unknown. BPE splits the difference: frequent words become single tokens, while rare or novel words break into familiar chunks. That is why “the” is one token but a strange name might be five.

This is the literal input layer of a large language model. Before the transformer does any thinking, before GPT-3 or ChatGPT predicts its next token, the text is turned into pieces exactly like these. It is also why models are billed per token and count them, not words: the token is the real unit of natural language processing here.

Type your own name, or a long rare word, and watch how many pieces it takes.

Missed the earlier editions? Train a perceptron, watch a Markov chain babble, lose to unbeatable tic-tac-toe, teach a filter to see edges, or watch an agent learn from reward.