What is a token in the context of LLMs?

Answer

A token is the basic unit of text that a language model reads and produces, typically a word fragment, whole word, or punctuation mark rather than a single character. A tokenizer splits input text into tokens and maps them to integer IDs the model can process. As a rough guide, one token is about four characters or three-quarters of a word in English, and model limits and pricing are usually measured in tokens.