Back to Tags
Tokenization
4 items tagged with "tokenization"
Filter by type:
FAQs2
FAQ
What is a token in the context of LLMs?
A token is the basic unit of text that a language model reads and produces, typically a word fragment, whole word, or punctuation mark rather than a s...
FAQ
What is a context window?
A context window is the maximum number of tokens a language model can consider at once, covering both the input prompt and the generated output. If a ...
Glossaries2
Glossary
Token
A token is the basic unit of text an LLM processes, typically a word fragment, whole word, or character, produced by a tokenizer and mapped to a numeric ID.
Glossary
Tokenization
Tokenization is the process of splitting raw text into tokens that a model can map to numeric IDs, usually using a subword algorithm such as byte-pair encoding.