NLP
15 items tagged with "nlp"
Benchmarks2
MTEB (Massive Text Embedding Benchmark)
A broad benchmark for text embedding models spanning classification, clustering, retrieval, reranking, and semantic similarity across many datasets and languages.
BEIR (Benchmarking Information Retrieval)
A heterogeneous zero-shot retrieval benchmark that tests how well a single retrieval model generalizes across diverse domains and query types without task-specific training.
FAQs6
What is a large language model (LLM)?
A large language model is a neural network trained on vast amounts of text to predict the next token in a sequence, which lets it generate and underst...
What are embeddings in machine learning?
Embeddings are dense numerical vectors that represent text, images, or other data in a continuous space where semantic similarity corresponds to geome...
What is a token in the context of LLMs?
A token is the basic unit of text that a language model reads and produces, typically a word fragment, whole word, or punctuation mark rather than a s...
What is prompt engineering?
Prompt engineering is the practice of designing the instructions, examples, and context given to a language model to get reliable, accurate outputs. T...
What is temperature in LLM generation?
Temperature is a parameter that controls the randomness of a language model's output by scaling the probability distribution over the next token. A lo...
What is chain-of-thought prompting?
Chain-of-thought prompting asks a language model to work through a problem step by step before giving a final answer, rather than responding immediate...
Glossaries6
Large Language Model (LLM)
A large language model is a neural network trained on vast text corpora to predict the next token, enabling it to generate and understand natural language across many tasks.
Token
A token is the basic unit of text an LLM processes, typically a word fragment, whole word, or character, produced by a tokenizer and mapped to a numeric ID.
Tokenization
Tokenization is the process of splitting raw text into tokens that a model can map to numeric IDs, usually using a subword algorithm such as byte-pair encoding.
Context Window
The context window is the maximum number of tokens a language model can consider at once, covering both the input prompt and the generated output.
Embedding
An embedding is a dense numeric vector that represents the meaning of text, an image, or other data so that similar items sit close together in vector space.
Prompt Engineering
Prompt engineering is the practice of designing and refining the text instructions given to a language model to steer its output toward accurate, useful results.