AI ML
226 items tagged with "ai-ml"
Best Practices15
Feature Store Best Practices
A centralized system for defining, storing, and serving machine learning features consistently for training and inference, avoiding skew and duplicated work.
MLOps Principles
The discipline of applying DevOps and engineering rigor to machine learning so models are built, deployed, monitored, and retrained reliably and reproducibly.
ML Model Monitoring and Drift Detection
Continuously tracking deployed ML models for performance decay, data drift, and concept drift so degradation is caught and corrected before it harms outcomes.
Data Version Control (DVC)
Versioning datasets, models, and ML pipelines alongside code so experiments are reproducible, using Git for metadata and external storage for large files.
Retrieval-Augmented Generation (RAG) Best Practices
RAG grounds a large language model in external documents retrieved at query time, reducing hallucination and letting answers reflect current, private data without retraining the model.
Prompt Engineering Best Practices
Prompt engineering is the practice of designing clear instructions, examples, and structure so a large language model returns accurate, consistent, and useful output.
LLM Evaluation and Evals
LLM evaluation measures model and application quality with repeatable tests, scoring accuracy, faithfulness, safety, and cost so teams can ship and improve with evidence.
LLM Guardrails
LLM guardrails are programmatic checks on model inputs and outputs that enforce safety, format, topic, and policy rules, blocking or correcting unsafe or off-policy responses.
Model Context Protocol (MCP)
The Model Context Protocol is an open standard that lets AI applications connect to external tools and data sources through a uniform client-server interface.
AI Agent Design Patterns
AI agent design patterns are reusable structures for LLM systems that plan, use tools, and act over multiple steps, covering reflection, tool use, planning, and multi-agent collaboration.
LLM Observability
LLM observability is the practice of tracing, logging, and measuring LLM applications in production to monitor quality, cost, latency, and safety and to debug failures.
Vector Database Best Practices
A vector database stores embeddings and serves fast similarity search for AI features like RAG and semantic search; best practices cover indexing, metadata, and freshness.
LLM Cost Optimization
LLM cost optimization reduces the spend of running language model applications through model selection, caching, prompt efficiency, and token-aware design without sacrificing quality.
Fine-Tuning vs RAG Decision Framework
A decision framework for choosing between fine-tuning, RAG, or both, based on whether the goal is new knowledge, consistent behavior, freshness, or domain adaptation.
Hallucination Mitigation
Hallucination mitigation reduces confident but false LLM output through grounding, retrieval, citation, verification, and uncertainty handling so answers can be trusted.
Tutorials12
How to set up vector similarity search with pgvector
Install pgvector in PostgreSQL, store embeddings, build an ANN index, and run nearest-neighbor queries for semantic search.
How to build a RAG pipeline for question answering
Build a retrieval-augmented generation pipeline that grounds an LLM's answers in your own documents using chunking, embeddings, and a vector store.
How to add semantic search with embeddings and a vector database
Add meaning-based search to an app by generating text embeddings and querying a vector database for nearest neighbors.
How to build an LLM app with function calling (tools)
Give an LLM the ability to call your functions, so it can fetch data and take actions instead of only producing text.
How to fine-tune a language model on your own data
Adapt a base language model to your domain with supervised fine-tuning, covering data prep, training, and evaluation.
How to evaluate LLM outputs systematically
Build a repeatable evaluation suite for LLM features using reference checks, rubrics, and model-graded scoring.
How to engineer effective prompts for LLMs
Apply practical prompt-engineering techniques such as clear instructions, examples, and structured output to get reliable LLM results.
How to build an AI agent that plans and acts
Build an autonomous LLM agent that uses tools in a perceive-plan-act loop to accomplish multi-step tasks.
How to deploy a machine learning model for inference
Serve a trained model behind an HTTP API, containerize it, and scale it for production inference.
How to add guardrails to an LLM application
Protect an LLM app with input and output guardrails that filter unsafe content, block prompt injection, and validate structure.
How to set up LLM observability and tracing
Instrument an LLM application to trace prompts, responses, tokens, latency, and cost so you can debug and optimize in production.
How to build a Model Context Protocol (MCP) server
Build an MCP server that exposes tools and resources to AI assistants over a standard protocol.
Blueprints6
Batch ML Scoring to Real-Time Model Serving Blueprint
Move from nightly batch ML scoring to low-latency online model serving with a model server, feature lookups, and autoscaling.
Self-Hosted LLM to Managed Inference Blueprint
Migrate self-hosted GPU LLM serving to a managed inference API to cut operational burden while controlling cost, latency, and data governance.
Notebooks to MLOps Pipeline Blueprint
Productionize ad-hoc data science notebooks into reproducible, versioned MLOps pipelines with CI/CD, tracking, and automated retraining.
Keyword Search to RAG Platform Blueprint
Build a retrieval-augmented generation platform over your corpus with embeddings, a vector database, and grounded LLM answers replacing keyword search.
Classic ML to Feature Store and Serving Blueprint
Migrate ad-hoc feature engineering for classic ML models to a central feature store with consistent offline training and online serving.
Ad-Hoc Prompt Scripts to AI Agent Platform Blueprint
Consolidate scattered LLM prompt scripts into a governed AI agent platform with tool use via MCP, guardrails, evals, and observability.
Reference Architectures21
ML Platform Architecture
End-to-end machine learning platform for training, deploying, and monitoring models
Feature Store Platform
A feature store on GCP providing consistent online and offline ML features with point-in-time correctness via Feast.
Vector Search Platform for RAG
A vector search platform on Azure powering retrieval-augmented generation with embeddings, a vector database, and hybrid search.
Retrieval-Augmented Generation Platform on AWS
A reference design for a production RAG system on AWS that grounds LLM answers in private documents using a managed vector store and Bedrock models.
LLM Gateway and Proxy on Kubernetes
A reference design for a self-hosted LLM gateway on Kubernetes that centralizes routing, rate limiting, cost tracking, and guardrails across multiple model providers.
Real-Time Model Serving on GCP
A reference design for low-latency online inference on GCP using Vertex AI endpoints, autoscaling, and a feature lookup path for sub-100ms predictions.
Batch Inference Pipeline on Azure
A reference design for scheduled, large-scale offline scoring on Azure using Synapse, Azure ML batch endpoints, and a data lake for inputs and results.
End-to-End MLOps Platform on Kubernetes
A reference design for a portable MLOps platform on Kubernetes covering pipelines, experiment tracking, a model registry, serving, and monitoring.
Feature Store and Online Serving on AWS
A reference design for a dual offline/online feature store on AWS that keeps training and serving features consistent and serves them at low latency.
Multi-Agent LLM System on Azure
A reference design for a multi-agent application on Azure where specialized LLM agents coordinate through an orchestrator and shared tools to complete complex tasks.
LLM Fine-Tuning Pipeline on GCP
A reference design for fine-tuning open LLMs on GCP using Vertex AI custom training, parameter-efficient methods, and an evaluation gate before deployment.
Vector Database Platform on Kubernetes
A reference design for a self-hosted, multi-tenant vector database platform on Kubernetes that powers semantic search and RAG retrieval at scale.
Recommendation System on AWS
A reference design for a two-stage recommender on AWS combining candidate retrieval and ranking, with streaming feedback and real-time serving.
Real-Time Fraud Detection on GCP
A reference design for streaming fraud detection on GCP that scores transactions in milliseconds using a feature store, rules, and an ML model.
Computer Vision Inference Pipeline on Azure
A reference design for an image and video computer-vision pipeline on Azure spanning ingestion, GPU inference, and human-in-the-loop review.
LLM Observability Stack on Kubernetes
A reference design for an LLM observability platform on Kubernetes that captures traces, evaluations, cost, and quality signals across AI applications.
AI Governance and Model Risk Platform (Multi-Cloud)
A reference design for a multi-cloud AI governance platform that inventories models, enforces policy, runs risk reviews, and maintains an audit trail.
Streaming ML Feature Pipeline on GCP
A reference design for a real-time feature engineering pipeline on GCP that computes streaming aggregates and serves them to online models consistently.
Self-Hosted Open LLM Inference on Kubernetes
A reference design for serving open-weight LLMs on Kubernetes with GPU autoscaling, continuous batching, and an OpenAI-compatible API.
Conversational AI Assistant on Azure
A reference design for an enterprise conversational assistant on Azure that combines RAG, tool calling, and memory behind a governed orchestration layer.
Data Labeling and Training Platform on AWS
A reference design for a closed-loop data labeling and training platform on AWS that turns raw data into labeled datasets and continuously retrains models.
Playbooks11
Feature Store Program Playbook
Build a feature store to unify ML feature engineering, ensure online and offline consistency, and enable feature reuse across teams.
MLOps Platform Build Playbook
A phased program to stand up an end-to-end MLOps platform covering feature stores, training pipelines, model registry, and automated deployment.
LLM and RAG Application Rollout Playbook
A program for rolling out a retrieval-augmented generation application from prototype to governed production with evals, guardrails, and observability.
Model Serving Migration Playbook
A phased program to migrate ML model serving from bespoke endpoints to a standardized, autoscaling, observable serving platform.
Responsible AI Governance Playbook
A program to establish responsible-AI governance covering risk assessment, controls, model documentation, and ongoing oversight aligned to NIST AI RMF and ISO 42001.
AI Red-Teaming Program Playbook
A phased program to build an AI red-teaming capability that adversarially tests LLM systems for jailbreaks, prompt injection, and harmful outputs.
Vector Database Adoption Playbook
A program to adopt a vector database for semantic search and RAG, covering embeddings, indexing, scaling, and operations.
Real-Time Inference Program Playbook
A program to deliver low-latency, high-throughput real-time ML inference with autoscaling, feature freshness, and strict SLOs.
LLM Evaluation Program Playbook
A program to build a rigorous LLM evaluation capability with offline evals, online metrics, and regression gating in CI.
Feature Store Rollout Playbook
A program to roll out a feature store that unifies feature engineering for training and serving with consistency and reuse.
AI Agent Platform Program Playbook
A program to build a governed AI agent platform with tool integration via MCP, guardrails, evaluation, and observability.
Checklists8
LLM/RAG Production-Readiness Checklist
Verification items for taking a retrieval-augmented generation (RAG) application from prototype to reliable production service.
ML Model Deployment Checklist
Pre-flight verification for promoting a trained machine learning model into a production serving environment.
MLOps Pipeline Review Checklist
Audit items for assessing the maturity, reproducibility, and automation of an end-to-end machine learning operations pipeline.
Responsible-AI Review Checklist
Governance verification items for assessing fairness, transparency, accountability, and risk before deploying an AI system.
AI Red-Team Checklist
Adversarial test items for probing an LLM or AI application for prompt injection, jailbreaks, data leakage, and unsafe behavior.
LLM Evaluation Readiness Checklist
Verification items for building a trustworthy evaluation harness before releasing changes to an LLM-powered feature.
AI Agent Deployment Checklist
Pre-flight items for safely deploying an autonomous LLM agent that calls tools and takes actions on behalf of users.
LLM Cost Optimization Review Checklist
Review items for reducing the cost of an LLM application without degrading quality, covering prompts, caching, and model choice.
Regulations7
Executive Order 14110 on Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence
US federal executive order directing agencies to set safety, security, and rights standards for AI development and deployment across government and industry.
Colorado Artificial Intelligence Act
Colorado state law requiring developers and deployers of high-risk AI systems to prevent algorithmic discrimination in consequential decisions.
EU AI Liability Directive (Proposed)
Proposed EU directive easing the burden of proof for victims claiming damage caused by AI systems, complementing the EU AI Act.
Interim Measures for the Management of Generative Artificial Intelligence Services
Chinese regulation governing public-facing generative AI services, covering content safety, training data, and security review.
Artificial Intelligence and Data Act
Proposed Canadian federal law (part of Bill C-27) regulating high-impact AI systems for safety and non-discrimination.
New York City Local Law 144 on Automated Employment Decision Tools
NYC law requiring bias audits and candidate notice for automated employment decision tools used in hiring and promotion.
ISO/IEC 42001 Artificial Intelligence Management System
International management-system standard for governing the responsible development and use of artificial intelligence within organizations.
Stacks19
MLOps Stack
MLflow, Kubeflow, TensorFlow, Feature Store - ML lifecycle
Databricks Lakehouse
Unified lakehouse architecture combining data-lake storage with warehouse reliability via Delta Lake, Spark compute, Unity Catalog governance, and ML.
MLflow MLOps Stack
End-to-end MLOps pattern using MLflow for experiment tracking, model registry, packaging, and deployment, integrated with feature, data, and serving layers.
RAG Stack (LangChain + pgvector + LLM)
Retrieval-augmented generation stack: LangChain orchestrates retrieval over pgvector embeddings in Postgres and grounds an LLM's answers in your own data.
Kubeflow ML Platform
Kubernetes-native ML platform: Kubeflow Pipelines, training operators, KServe serving, and Katib tuning run the ML lifecycle on Kubernetes.
Feast Feature Store Stack
Feature store pattern using Feast to define, materialize, and serve consistent ML features from an offline warehouse and a low-latency online store.
LLM Agent Stack
Agentic AI stack: an orchestration framework drives an LLM to reason, call tools and APIs, and use vector memory to complete multi-step tasks autonomously.
React + Django REST Framework
A React frontend consuming a Django REST Framework API, a productive Python full-stack for data-rich and ML-adjacent web applications.
Ray Distributed ML
A unified compute stack using Ray to scale Python machine learning workloads from data processing through training to serving.
vLLM + Ray Serve
A high-throughput LLM serving stack combining vLLM's optimized inference engine with Ray Serve for scalable, multi-replica deployment.
LangGraph + pgvector
An agentic LLM application stack using LangGraph for stateful agent workflows backed by Postgres with pgvector for retrieval and memory.
Haystack RAG
A production RAG stack built on the Haystack framework for composable retrieval-augmented generation pipelines over a document store.
LlamaIndex + Qdrant
A RAG and data-framework stack pairing LlamaIndex for LLM data ingestion and querying with Qdrant as a high-performance vector database.
Amazon Bedrock RAG
A managed RAG stack on AWS using Amazon Bedrock foundation models with Knowledge Bases for retrieval over data stored in S3.
Vertex AI Pipeline
A managed MLOps stack on Google Cloud using Vertex AI Pipelines to orchestrate training, evaluation, and deployment of ML models.
Amazon SageMaker MLOps
A managed MLOps stack on AWS using Amazon SageMaker to build, train, deploy, and monitor machine learning models end to end.
Triton Inference Server
A high-performance model-serving stack using NVIDIA Triton to serve models from any framework with GPU optimization on Kubernetes.
Tecton Feature Store
A production feature platform stack using Tecton to define, compute, and serve consistent ML features for training and real-time inference.
MosaicML Composer Training
A large-scale model training stack using Composer and the MosaicML toolkit to train and fine-tune models efficiently on GPU clusters.
Comparisons27
Databricks vs Snowflake
Lakehouse platform built on Spark and Delta Lake versus Snowflake's cloud data warehouse, now converging on shared use cases.
pgvector vs Pinecone
A Postgres extension that adds vector search to your existing database versus a fully managed, purpose-built vector database.
Python vs R
Two leading languages for data work: Python is a general-purpose language strong across the data and ML pipeline, while R is purpose-built for statistics and visualization.
Ruby vs Python
Two expressive, dynamic languages: Ruby is beloved for web development with Rails, while Python dominates data, ML, and general-purpose scripting.
RAG vs Fine-Tuning
Retrieval-augmented generation injects external knowledge at query time; fine-tuning bakes behavior into model weights. They solve different problems and often combine.
Fine-Tuning vs Prompt Engineering
Prompt engineering steers a model with instructions and examples in the context; fine-tuning changes the weights. Cost, control, and durability differ sharply.
GPT vs Claude
OpenAI's GPT and Anthropic's Claude are leading proprietary LLM families. They differ in design philosophy, context handling, and integration ecosystems rather than raw capability tier.
Open-Weight vs Proprietary LLMs
Open-weight models can be downloaded and self-hosted; proprietary models are accessed via API. The split shapes control, cost, privacy, and capability ceilings.
Llama vs Mistral
Llama (Meta) and Mistral are two leading open-weight LLM families. They differ in licensing, model sizes, mixture-of-experts use, and ecosystem maturity.
vLLM vs TGI
vLLM and Hugging Face Text Generation Inference (TGI) are high-throughput LLM serving engines. Both optimize GPU inference but differ in ecosystem and tuning focus.
LangChain vs LlamaIndex
LangChain is a broad framework for LLM application orchestration; LlamaIndex specializes in data ingestion and retrieval for RAG. They overlap but optimize for different centers of gravity.
Pinecone vs Weaviate
Pinecone is a fully managed vector database; Weaviate is open source with managed and self-hosted options. The split shapes control, operations, and feature flexibility.
Weaviate vs Qdrant
Weaviate and Qdrant are open-source vector databases. Weaviate emphasizes a modular, feature-rich platform; Qdrant emphasizes a lean, performant Rust-based engine.
pgvector vs Dedicated Vector Database
pgvector adds vector search to PostgreSQL; a dedicated vector database is purpose-built. The choice trades operational simplicity against scale and specialized features.
Embeddings vs Keyword Search
Embedding-based semantic search matches by meaning; keyword search matches by terms. Each handles different query types, and hybrid search often beats either alone.
PyTorch vs TensorFlow
PyTorch and TensorFlow are the two dominant deep-learning frameworks. PyTorch leads in research and flexibility; TensorFlow has strong production and deployment tooling.
Hugging Face vs OpenAI API
Hugging Face provides open models, tooling, and self-hosting paths; the OpenAI API offers managed access to proprietary models. The choice trades control against convenience.
Batch vs Real-Time Inference
Batch inference processes data in scheduled bulk jobs; real-time inference serves predictions on demand. They trade latency against throughput, cost, and complexity.
CPU vs GPU Inference
CPUs and GPUs both run ML inference. GPUs excel at parallel, large-model workloads; CPUs are cheaper and simpler for small models and low concurrency.
Quantization vs Full Precision
Quantization stores model weights at lower bit-widths to cut memory and speed inference; full precision preserves maximum accuracy. The trade-off is size and speed versus fidelity.
AI Agents vs Workflows
Agentic systems let an LLM decide its own steps and tool use; workflows orchestrate LLMs through predefined paths. The choice trades flexibility against predictability.
MLflow vs Weights & Biases
MLflow is an open-source ML lifecycle platform; Weights & Biases is a polished experiment-tracking SaaS. The choice trades self-hosted breadth against managed experience.
Kubeflow vs SageMaker
Kubeflow is an open-source ML platform on Kubernetes; Amazon SageMaker is AWS's managed ML service. The choice trades portability and control against managed convenience.
Feature Store: Build vs Buy
Teams can build a custom feature store or adopt a managed or open-source one. The choice trades control and fit against time-to-value and maintenance burden.
Self-Hosted vs Managed Inference
Self-hosted inference runs models on your own infrastructure; managed inference uses a provider's endpoint. The choice trades control, privacy, and cost-at-scale against simplicity.
DVC vs Git LFS
DVC and Git LFS both version large files alongside Git. DVC targets ML data and pipelines; Git LFS is a general-purpose large-file extension.
Data Lake vs Data Warehouse
A data lake stores raw data of any type cheaply; a data warehouse stores structured, modeled data for fast analytics. They serve different stages and users.
Benchmarks57
Code Translation Accuracy
Measures how accurately AI models translate code between programming languages
MMLU (Massive Multitask Language Understanding)
A 57-subject multiple-choice benchmark testing broad academic and professional knowledge across STEM, humanities, social sciences, and law.
MMLU-Pro
A harder, reasoning-focused successor to MMLU with ten answer options and tougher questions designed to separate frontier models that saturated the original.
GSM8K (Grade School Math 8K)
A benchmark of ~8,500 grade-school math word problems that test multi-step arithmetic reasoning with a single numeric answer.
MATH (Competition Mathematics)
A benchmark of 12,500 competition-style math problems across algebra, geometry, number theory, and calculus, graded on exact final-answer match.
HumanEval
A code-generation benchmark of 164 Python programming problems graded by executing unit tests, popularizing the pass@k metric.
MBPP (Mostly Basic Python Problems)
A benchmark of ~1,000 entry-level Python programming tasks with test cases, used to evaluate basic code synthesis from short descriptions.
SWE-bench
A benchmark of real GitHub issues from open-source Python repositories where a model must produce a patch that resolves the issue and passes tests.
SWE-bench Verified
A 500-task, human-validated subset of SWE-bench with clear specifications and reliable tests, used as the standard clean measure of agentic coding.
BIG-bench (Beyond the Imitation Game)
A collaborative suite of 200+ diverse tasks probing reasoning, knowledge, and emergent abilities beyond conventional language benchmarks.
BBH (BIG-bench Hard)
A 23-task subset of BIG-bench focused on challenging multi-step reasoning where chain-of-thought prompting yields large gains.
HellaSwag
A commonsense sentence-completion benchmark where models pick the most plausible continuation among adversarially generated distractors.
ARC (AI2 Reasoning Challenge)
A grade-school science question benchmark split into Easy and Challenge sets, the latter built from questions retrieval methods answer incorrectly.
TruthfulQA
A benchmark measuring whether models avoid generating false answers that mimic common human misconceptions and falsehoods.
GPQA (Graduate-Level Google-Proof Q&A)
A benchmark of expert-written, graduate-level science questions designed to be extremely hard even with web access, testing deep domain reasoning.
MMMU (Massive Multi-discipline Multimodal Understanding)
A multimodal benchmark of college-level questions requiring joint reasoning over text and images such as diagrams, charts, and figures.
MT-Bench
A multi-turn conversational benchmark where a strong LLM judge scores model responses across categories on a 1-10 quality scale.
Chatbot Arena
A live, crowdsourced evaluation where users compare two anonymous model responses and votes are aggregated into Elo-style rankings.
HELM (Holistic Evaluation of Language Models)
A standardized framework evaluating language models across many scenarios and multiple metrics including accuracy, robustness, fairness, and efficiency.
DROP (Discrete Reasoning Over Paragraphs)
A reading-comprehension benchmark requiring discrete operations like addition, counting, sorting, and comparison over passage content.
WinoGrande
A large-scale commonsense benchmark of pronoun-resolution sentence pairs designed to require world knowledge rather than lexical cues.
AGIEval
A benchmark built from human standardized exams such as college entrance, law, and civil-service tests to measure human-centric reasoning.
LiveCodeBench
A contamination-resistant coding benchmark that continuously collects new competitive-programming problems and evaluates by execution over time.
AIME (Competition Math Benchmark)
An olympiad-level math benchmark using American Invitational Mathematics Examination problems with integer answers, a key frontier reasoning test.
tau-bench (Tool-Agent-User Benchmark)
An agentic benchmark testing tool-using models in simulated customer-service dialogues that require following domain policies and calling APIs correctly.
Terminal-Bench
An agentic benchmark evaluating models on completing real command-line tasks inside a sandboxed terminal, verified by automated checks.
RULER (Long-Context Benchmark)
A synthetic long-context benchmark with configurable tasks measuring a model's effective context length beyond simple retrieval.
TPCx-AI
An end-to-end machine-learning benchmark measuring the full data-science pipeline — ingestion, training, and serving — across multiple AI use cases.
ANN-Benchmarks
The standard open benchmark for approximate nearest neighbor search, plotting recall against queries-per-second across vector index libraries and databases.
BigANN Benchmark
A billion-scale approximate nearest neighbor benchmark testing vector search algorithms on large data sets with constraints on memory, throughput, and recall.
MTEB (Massive Text Embedding Benchmark)
A broad benchmark for text embedding models spanning classification, clustering, retrieval, reranking, and semantic similarity across many datasets and languages.
BEIR (Benchmarking Information Retrieval)
A heterogeneous zero-shot retrieval benchmark that tests how well a single retrieval model generalizes across diverse domains and query types without task-specific training.
IFEval (Instruction-Following Eval)
A benchmark that measures whether LLMs follow precise, verifiable formatting and content instructions using automatic checks rather than subjective judgment.
BFCL (Berkeley Function-Calling Leaderboard)
A benchmark for evaluating how accurately LLMs select, call, and parameterize functions and tools, including parallel, multiple, and multi-turn calling scenarios.
MGSM (Multilingual Grade School Math)
A multilingual extension of grade-school math word problems that tests whether LLMs can reason through arithmetic in many languages, not just English.
SimpleQA
A factuality benchmark of short, fact-seeking questions with single verifiable answers, designed to measure how often LLMs are correct, wrong, or appropriately abstain.
FRAMES (Factuality, Retrieval, And reasoning MEasurement Set)
A benchmark for retrieval-augmented generation that tests end-to-end factuality, multi-document retrieval, and multi-hop reasoning on questions needing several sources.
MuSR (Multistep Soft Reasoning)
A benchmark of long natural-language narratives requiring multistep commonsense and logical reasoning, such as murder mysteries and object-placement puzzles.
LiveBench
A contamination-resistant benchmark that continuously refreshes questions from recent sources and grades automatically against objective ground truth across many task categories.
BigCodeBench
A code-generation benchmark for realistic programming tasks that require composing many library calls, evaluated with rigorous test suites and high branch coverage.
CRUXEval (Code Reasoning, Understanding, and Execution)
A benchmark that tests whether models can reason about code execution by predicting function inputs from outputs and outputs from inputs.
RewardBench
A benchmark for reward models and LLM judges that measures how well they prefer better responses over worse ones across chat, reasoning, safety, and refusal cases.
AlpacaEval
An automated LLM-as-judge benchmark that estimates a model's win rate against a reference model on open-ended instructions, with a length-controlled variant to reduce verbosity bias.
Arena-Hard
An automatic benchmark of challenging, real-user-derived prompts graded by an LLM judge, built to align closely with human preference rankings and separate strong models.
ImageNet (ILSVRC Classification)
The foundational large-scale image classification benchmark covering 1,000 object categories, long used to track progress in computer vision and pretraining.
COCO (Object Detection and Segmentation)
A large-scale benchmark for object detection, instance segmentation, and keypoints in complex everyday scenes, evaluated with mean average precision across IoU thresholds.
VQAv2 (Visual Question Answering)
A benchmark that tests whether models can answer open-ended natural-language questions about images, balanced to reduce language-only shortcuts.
MMBench (Multimodal Benchmark)
A systematic multimodal benchmark that evaluates vision-language models across many fine-grained ability dimensions using a robustness-checked multiple-choice protocol.
DocVQA (Document Visual Question Answering)
A benchmark for answering questions about document images, testing OCR, layout understanding, and reasoning over text, tables, and forms.
ChartQA
A benchmark for answering questions about charts and plots that require visual data extraction plus arithmetic and logical reasoning over the extracted values.
LibriSpeech (ASR Word Error Rate)
A widely used benchmark for automatic speech recognition built from read English audiobooks, measured primarily by word error rate on clean and noisy splits.
HarmBench
A standardized red-teaming benchmark that measures how often automated attacks elicit harmful behaviors from LLMs and how well refusal and defenses hold up.
MLPerf Training
Industry-standard benchmark suite measuring how fast hardware and software systems train machine-learning models to a fixed target quality.
MLPerf Inference
Benchmark suite measuring how fast and efficiently systems serve trained ML models under realistic latency and throughput constraints.
MLPerf Tiny
Benchmark suite for ultra-low-power machine learning on microcontrollers and embedded devices, measuring latency, energy, and accuracy.
DAWNBench
Stanford benchmark that measured end-to-end deep-learning training and inference by time-to-accuracy and cost, popularizing those metrics.
HPL-AI / HPL-MxP
Mixed-precision LINPACK variant measuring supercomputer performance using low-precision arithmetic refined to full accuracy, reflecting AI hardware.
FAQs20
What is a vector database?
A vector database stores and searches high-dimensional vector embeddings, the numeric representations of text, images, or other data produced by machi...
What is a large language model (LLM)?
A large language model is a neural network trained on vast amounts of text to predict the next token in a sequence, which lets it generate and underst...
What is retrieval-augmented generation (RAG)?
Retrieval-augmented generation is a technique that supplements a language model with relevant documents fetched at query time, rather than relying onl...
What are embeddings in machine learning?
Embeddings are dense numerical vectors that represent text, images, or other data in a continuous space where semantic similarity corresponds to geome...
What is the difference between fine-tuning and RAG?
Fine-tuning updates a model's weights by training it further on domain-specific examples, changing how the model behaves and what style or skills it h...
What is fine-tuning a model?
Fine-tuning is the process of continuing to train a pre-trained model on a smaller, task-specific dataset so it adapts to a particular domain, style, ...
What is a token in the context of LLMs?
A token is the basic unit of text that a language model reads and produces, typically a word fragment, whole word, or punctuation mark rather than a s...
What is a context window?
A context window is the maximum number of tokens a language model can consider at once, covering both the input prompt and the generated output. If a ...
What is prompt engineering?
Prompt engineering is the practice of designing the instructions, examples, and context given to a language model to get reliable, accurate outputs. T...
What is an AI agent?
An AI agent is a system that uses a language model to decide and take actions toward a goal, rather than producing a single response. It typically ope...
What is hallucination in LLMs?
Hallucination is when a language model produces text that sounds confident and plausible but is factually wrong or unsupported by its sources. It happ...
What is temperature in LLM generation?
Temperature is a parameter that controls the randomness of a language model's output by scaling the probability distribution over the next token. A lo...
What is inference in machine learning?
Inference is the phase where a trained model is used to make predictions or generate output on new inputs, as opposed to training where the model lear...
What is the difference between supervised and unsupervised learning?
Supervised learning trains a model on labeled examples, where each input has a known target, so the model learns to predict labels for new data in tas...
What is overfitting in machine learning?
Overfitting happens when a model learns the training data too closely, including its noise and quirks, so it performs well on that data but poorly on ...
What is a transformer architecture?
The transformer is a neural network architecture, introduced in 2017, that processes sequences using a self-attention mechanism instead of recurrence....
What is quantization in machine learning?
Quantization reduces the numerical precision of a model's weights and activations, for example from 32-bit floating point to 8-bit or 4-bit integers, ...
What is the Model Context Protocol (MCP)?
The Model Context Protocol is an open standard that defines how AI applications connect to external tools, data sources, and services in a consistent ...
What is chain-of-thought prompting?
Chain-of-thought prompting asks a language model to work through a problem step by step before giving a final answer, rather than responding immediate...
What is responsible AI?
Responsible AI is the practice of designing, building, and operating AI systems so they are fair, transparent, accountable, secure, and respectful of ...
Glossaries21
Data Lake
A data lake is a centralized repository that stores large volumes of raw data in its native format — structured, semi-structured, and unstructured — at low cost, with schema applied at read time rather than on ingestion.
Large Language Model (LLM)
A large language model is a neural network trained on vast text corpora to predict the next token, enabling it to generate and understand natural language across many tasks.
Token
A token is the basic unit of text an LLM processes, typically a word fragment, whole word, or character, produced by a tokenizer and mapped to a numeric ID.
Tokenization
Tokenization is the process of splitting raw text into tokens that a model can map to numeric IDs, usually using a subword algorithm such as byte-pair encoding.
Context Window
The context window is the maximum number of tokens a language model can consider at once, covering both the input prompt and the generated output.
Embedding
An embedding is a dense numeric vector that represents the meaning of text, an image, or other data so that similar items sit close together in vector space.
Vector Search
Vector search finds items whose embeddings are closest to a query embedding, enabling semantic retrieval by meaning rather than exact keyword match.
Retrieval-Augmented Generation (RAG)
Retrieval-augmented generation is a technique that retrieves relevant documents at query time and supplies them to an LLM as context so its answers are grounded in external data.
Fine-Tuning
Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset to specialize its behavior, style, or domain knowledge.
Prompt Engineering
Prompt engineering is the practice of designing and refining the text instructions given to a language model to steer its output toward accurate, useful results.
Inference
Inference is the process of running a trained model on new inputs to produce outputs, as opposed to the training phase that creates the model.
Training
Training is the process of adjusting a model's parameters from data so it learns to perform a task, typically by minimizing a loss function with gradient descent.
Transformer
A transformer is a neural network architecture built around self-attention that processes sequences in parallel, forming the basis of modern large language models.
Attention Mechanism
An attention mechanism lets a model weigh the relevance of different parts of its input when producing each output, focusing on the most pertinent tokens.
Hallucination
Hallucination is when a language model generates fluent, confident output that is factually incorrect, fabricated, or unsupported by its sources.
Temperature (LLM Sampling)
Temperature is a sampling parameter that scales an LLM's output probabilities, controlling how random or deterministic its token choices are.
Top-p Sampling (Nucleus Sampling)
Top-p sampling restricts an LLM's next-token choice to the smallest set of tokens whose cumulative probability exceeds a threshold p, then samples from that set.
Quantization
Quantization reduces the numeric precision of a model's weights and activations, shrinking memory use and speeding inference with limited accuracy loss.
AI Agent
An AI agent is a system that uses a language model to plan and take actions toward a goal, calling tools, observing results, and iterating with limited human input.
Model Context Protocol (MCP)
The Model Context Protocol is an open standard that defines how AI applications connect to external tools, data sources, and prompts through a uniform interface.
Foundation Model
A foundation model is a large model pretrained on broad data at scale that can be adapted, through fine-tuning or prompting, to a wide range of downstream tasks.