Tokenization
Tokenization splits text into subword tokens using algorithms like byte-pair encoding, controlling cost, context usage, and model quality.
Tokenization is the step that converts raw text into a sequence of tokens before a model can process it. Modern language models rely on subword tokenization, which balances vocabulary size against the ability to represent any string, including rare words and typos, by composing them from smaller pieces.
How It Works
A tokenizer is trained on a corpus to learn a vocabulary of frequent character sequences. Byte-pair encoding (BPE) and similar methods start from individual bytes or characters and repeatedly merge the most common adjacent pairs until the vocabulary reaches a target size. At runtime the tokenizer greedily applies these merges to split input text, then maps each resulting token to an integer ID. The reverse process, detokenization, reconstructs text from IDs.
Why It Matters
Tokenization determines how many tokens a given text consumes, which controls cost and how much content fits in a model's context window. It also shapes model behavior: text that tokenizes poorly, such as code, long numbers, or non-English scripts, can degrade quality or inflate token counts. Using the exact tokenizer that matches a target model is important when estimating limits and chunking documents.
Related Terms
Tokenization produces token sequences that fill the context-window and are turned into embedding vectors before an large-language-model runs.