Token

A token is the smallest piece of text that a language model reads or generates. Tokens are usually subword fragments rather than whole words, so a common word may be one token while a rare or long word splits into several. Each token maps to an integer ID drawn from the model's fixed vocabulary.

How It Works

Before a model sees text, a tokenizer splits the string into tokens using an algorithm such as byte-pair encoding. Each token ID is then looked up in an embedding table to produce a vector. As a rough guide for English, one token is about four characters or three-quarters of a word, though this varies by language and content. The model also emits text token by token during generation.

Why It Matters

Tokens are the unit of measurement for both context windows and pricing, so token counts directly affect cost and how much text fits in a single request. Understanding tokenization helps explain odd behavior, such as poor handling of unusual spellings, long numbers, or non-English scripts that tokenize inefficiently. Counting tokens accurately is essential when chunking documents for retrieval or staying within model limits.

Related Terms

Tokens are produced by tokenization, consumed within a context-window, and converted into vectors as embeddings before an large-language-model processes them.