Benchmark
124 items tagged with "benchmark"
Benchmarks124
Migration Plan Generation Quality
Evaluates the quality and completeness of AI-generated migration plans
Documentation Generation Quality
Evaluates AI-generated documentation for migrated systems
MMLU (Massive Multitask Language Understanding)
A 57-subject multiple-choice benchmark testing broad academic and professional knowledge across STEM, humanities, social sciences, and law.
MMLU-Pro
A harder, reasoning-focused successor to MMLU with ten answer options and tougher questions designed to separate frontier models that saturated the original.
GSM8K (Grade School Math 8K)
A benchmark of ~8,500 grade-school math word problems that test multi-step arithmetic reasoning with a single numeric answer.
MATH (Competition Mathematics)
A benchmark of 12,500 competition-style math problems across algebra, geometry, number theory, and calculus, graded on exact final-answer match.
HumanEval
A code-generation benchmark of 164 Python programming problems graded by executing unit tests, popularizing the pass@k metric.
MBPP (Mostly Basic Python Problems)
A benchmark of ~1,000 entry-level Python programming tasks with test cases, used to evaluate basic code synthesis from short descriptions.
SWE-bench
A benchmark of real GitHub issues from open-source Python repositories where a model must produce a patch that resolves the issue and passes tests.
SWE-bench Verified
A 500-task, human-validated subset of SWE-bench with clear specifications and reliable tests, used as the standard clean measure of agentic coding.
BIG-bench (Beyond the Imitation Game)
A collaborative suite of 200+ diverse tasks probing reasoning, knowledge, and emergent abilities beyond conventional language benchmarks.
BBH (BIG-bench Hard)
A 23-task subset of BIG-bench focused on challenging multi-step reasoning where chain-of-thought prompting yields large gains.
HellaSwag
A commonsense sentence-completion benchmark where models pick the most plausible continuation among adversarially generated distractors.
ARC (AI2 Reasoning Challenge)
A grade-school science question benchmark split into Easy and Challenge sets, the latter built from questions retrieval methods answer incorrectly.
TruthfulQA
A benchmark measuring whether models avoid generating false answers that mimic common human misconceptions and falsehoods.
GPQA (Graduate-Level Google-Proof Q&A)
A benchmark of expert-written, graduate-level science questions designed to be extremely hard even with web access, testing deep domain reasoning.
MMMU (Massive Multi-discipline Multimodal Understanding)
A multimodal benchmark of college-level questions requiring joint reasoning over text and images such as diagrams, charts, and figures.
MT-Bench
A multi-turn conversational benchmark where a strong LLM judge scores model responses across categories on a 1-10 quality scale.
Chatbot Arena
A live, crowdsourced evaluation where users compare two anonymous model responses and votes are aggregated into Elo-style rankings.
HELM (Holistic Evaluation of Language Models)
A standardized framework evaluating language models across many scenarios and multiple metrics including accuracy, robustness, fairness, and efficiency.
DROP (Discrete Reasoning Over Paragraphs)
A reading-comprehension benchmark requiring discrete operations like addition, counting, sorting, and comparison over passage content.
WinoGrande
A large-scale commonsense benchmark of pronoun-resolution sentence pairs designed to require world knowledge rather than lexical cues.
AGIEval
A benchmark built from human standardized exams such as college entrance, law, and civil-service tests to measure human-centric reasoning.
LiveCodeBench
A contamination-resistant coding benchmark that continuously collects new competitive-programming problems and evaluates by execution over time.
AIME (Competition Math Benchmark)
An olympiad-level math benchmark using American Invitational Mathematics Examination problems with integer answers, a key frontier reasoning test.
tau-bench (Tool-Agent-User Benchmark)
An agentic benchmark testing tool-using models in simulated customer-service dialogues that require following domain policies and calling APIs correctly.
Terminal-Bench
An agentic benchmark evaluating models on completing real command-line tasks inside a sandboxed terminal, verified by automated checks.
RULER (Long-Context Benchmark)
A synthetic long-context benchmark with configurable tasks measuring a model's effective context length beyond simple retrieval.
TPC-C
The classic OLTP benchmark simulating an order-entry warehouse workload, measuring transactions per minute (tpmC) and price/performance.
TPC-H
An ad-hoc decision-support benchmark of 22 complex analytical queries over a star-like schema, reporting query throughput and power at fixed scale factors.
TPC-DS
A modern decision-support benchmark with 99 queries over a multi-snowflake retail schema, designed to stress complex analytics, data loading, and concurrency.
TPC-E
An OLTP benchmark modeling a brokerage firm with realistic data and read-heavy transactions, reporting tpsE as a more representative successor to TPC-C.
YCSB
The Yahoo! Cloud Serving Benchmark for NoSQL and key-value stores, defining standard read/write workload mixes to compare throughput and latency.
sysbench
A scriptable, multi-threaded benchmark tool widely used for MySQL/PostgreSQL OLTP tests as well as CPU, memory, and file I/O microbenchmarks.
pgbench
PostgreSQL's built-in benchmarking tool that runs a TPC-B-like transaction load and custom scripts to measure transactions per second and latency.
HammerDB
An open-source database load-testing tool implementing TPROC-C and TPROC-H (TPC-C/TPC-H-derived) workloads across many relational engines.
BenchBase
An extensible Java framework (successor to OLTPBench) for benchmarking relational databases with many built-in workloads via a common JDBC harness.
ClickBench
An open benchmark for analytical databases using a single wide web-analytics table and 43 queries to compare cold and hot OLAP query latency.
Star Schema Benchmark (SSB)
A simplified TPC-H derivative using a classic star schema and 13 queries in four flights to measure data-warehouse query performance.
TPCx-BB (BigBench)
An end-to-end big-data analytics benchmark mixing SQL, machine learning, and natural-language processing over a retail data set on Hadoop/Spark-style platforms.
TPCx-AI
An end-to-end machine-learning benchmark measuring the full data-science pipeline — ingestion, training, and serving — across multiple AI use cases.
JMH (Java Microbenchmark Harness)
The standard harness for writing reliable JVM microbenchmarks, widely used to measure data-processing and serialization library performance on the JVM.
TeraSort
A distributed sort benchmark that orders one terabyte (or more) of data on a cluster, measuring big-data engine throughput and shuffle efficiency.
Sort Benchmark
A long-running family of competitive sorting benchmarks (GraySort, MinuteSort, JouleSort) that rank systems on speed, cost, and energy for large-scale sorting.
dbt Pipeline Performance Benchmark
A category of benchmarks measuring transformation pipeline performance in dbt — model build time, warehouse compute cost, and incremental run efficiency.
OLTP vs OLAP Latency Benchmark
A comparative benchmark category contrasting transactional (OLTP) point-operation latency with analytical (OLAP) scan-and-aggregate latency to evaluate HTAP systems.
ANN-Benchmarks
The standard open benchmark for approximate nearest neighbor search, plotting recall against queries-per-second across vector index libraries and databases.
BigANN Benchmark
A billion-scale approximate nearest neighbor benchmark testing vector search algorithms on large data sets with constraints on memory, throughput, and recall.
Time-Series Ingestion Benchmark
A benchmark category measuring write throughput, query latency, and compression for time-series databases under high-cardinality metric and event ingestion.
Streaming Throughput Benchmark
A benchmark category for event-streaming platforms measuring producer/consumer throughput and end-to-end latency under sustained load and varying durability settings.
In-Memory Store Benchmark (memtier/redis-benchmark)
Benchmarks for in-memory data stores like Redis and Memcached, measuring operations per second and sub-millisecond latency under varied key/value and pipeline settings.
ETL/ELT Pipeline Throughput Benchmark
A benchmark category measuring data-integration pipeline performance — extraction and load throughput, transformation latency, and end-to-end freshness.
fio (Flexible I/O Tester)
The standard tool for benchmarking storage I/O, measuring IOPS, bandwidth, and latency across configurable read/write patterns, block sizes, and queue depths.
LDBC Social Network Benchmark
The standard benchmark for graph databases, measuring interactive transactional and analytical query performance over a realistic, correlated social-network graph.
SPEC CPU 2017
Industry-standard CPU benchmark suite measuring integer and floating-point compute performance under realistic, compute-bound workloads.
SPECjbb 2015
Java server benchmark modeling a supermarket company's transaction processing to measure JVM and server-side Java throughput and latency.
CoreMark
Compact, portable CPU benchmark from EEMBC designed to measure embedded and microcontroller core performance with a single comparable number.
Dhrystone
Classic synthetic integer benchmark that produces DMIPS, a historical and still-cited measure of general-purpose integer CPU performance.
Whetstone
Historic synthetic floating-point benchmark measuring scientific-style arithmetic performance, reported in MWIPS (millions of Whetstone instructions per second).
STREAM
Simple, portable benchmark measuring sustainable main-memory bandwidth for large vector operations, the standard metric for memory-bound performance.
Geekbench
Cross-platform benchmark measuring single-core and multi-core CPU performance plus GPU compute, widely used to compare phones, laptops, and servers.
LINPACK / HPL
Dense linear-algebra benchmark solving a large system of equations to measure peak floating-point throughput; HPL ranks the TOP500 supercomputers.
TechEmpower Web Framework Benchmarks
Open benchmark suite comparing web frameworks and platforms across standardized request types like JSON, single-query, and plaintext throughput.
wrk HTTP Benchmark
Modern, multithreaded HTTP load-testing tool that generates high request volume from a single machine and reports throughput and latency distribution.
k6 Load Testing
Developer-centric, scriptable load-testing tool using JavaScript scenarios to measure API and web performance with rich thresholds and metrics.
Apache JMeter
Mature, GUI-driven Java load-testing tool for simulating complex multi-protocol user scenarios and measuring throughput, latency, and error rates.
Gatling Load Testing
Scala-based, asynchronous load-testing tool with an expressive scenario DSL and detailed HTML reports for high-concurrency performance testing.
Locust Load Testing
Python-based, distributed load-testing tool where user behavior is defined in code, scaling to many workers for high-concurrency scenario testing.
ApacheBench (ab)
Simple, ubiquitous command-line HTTP benchmarking tool for quick single-endpoint throughput and latency measurement, bundled with Apache.
fio Storage I/O Benchmark
Flexible I/O tester for measuring storage device and filesystem performance across configurable read/write patterns, block sizes, and queue depths.
iperf Network Benchmark
Active network measurement tool that generates TCP, UDP, and SCTP traffic between two hosts to measure achievable bandwidth, jitter, and packet loss.
netperf Network Benchmark
Network performance tool measuring both bulk-transfer throughput and request/response transaction rates, used to characterize latency-sensitive workloads.
Phoronix Test Suite
Open-source, cross-platform benchmarking framework that automates hundreds of real-world tests and aggregates results for reproducible comparison.
Core Web Vitals
Google's set of user-centric web performance metrics, LCP, INP, and CLS, that quantify loading, interactivity, and visual stability of real page loads.
Lighthouse Performance
Google's open-source web auditing tool that runs synthetic page loads and produces a 0-100 performance score from lab metrics like LCP, TBT, and CLS.
Cold-Start Latency Benchmark
Measures the added latency when a serverless function or container must initialize from scratch before serving its first request after being idle.
Build-Time Benchmark
Measures how long it takes to compile and package software, a key developer-productivity and CI-cost metric across clean, incremental, and cached builds.
Container Startup Time Benchmark
Measures how quickly a container goes from launch to ready, covering image pull, runtime creation, and application readiness for scaling and resilience.
API P99 Latency Benchmark
Measures tail latency, the response time at the 99th percentile, to capture worst-case API responsiveness that averages hide and that users feel most.
HTTP/3 and QUIC Protocol Benchmark
Measures how the QUIC-based HTTP/3 transport compares to HTTP/2 over TCP on connection setup, throughput, and latency, especially on lossy networks.
MTEB (Massive Text Embedding Benchmark)
A broad benchmark for text embedding models spanning classification, clustering, retrieval, reranking, and semantic similarity across many datasets and languages.
BEIR (Benchmarking Information Retrieval)
A heterogeneous zero-shot retrieval benchmark that tests how well a single retrieval model generalizes across diverse domains and query types without task-specific training.
IFEval (Instruction-Following Eval)
A benchmark that measures whether LLMs follow precise, verifiable formatting and content instructions using automatic checks rather than subjective judgment.
BFCL (Berkeley Function-Calling Leaderboard)
A benchmark for evaluating how accurately LLMs select, call, and parameterize functions and tools, including parallel, multiple, and multi-turn calling scenarios.
MGSM (Multilingual Grade School Math)
A multilingual extension of grade-school math word problems that tests whether LLMs can reason through arithmetic in many languages, not just English.
SimpleQA
A factuality benchmark of short, fact-seeking questions with single verifiable answers, designed to measure how often LLMs are correct, wrong, or appropriately abstain.
FRAMES (Factuality, Retrieval, And reasoning MEasurement Set)
A benchmark for retrieval-augmented generation that tests end-to-end factuality, multi-document retrieval, and multi-hop reasoning on questions needing several sources.
MuSR (Multistep Soft Reasoning)
A benchmark of long natural-language narratives requiring multistep commonsense and logical reasoning, such as murder mysteries and object-placement puzzles.
LiveBench
A contamination-resistant benchmark that continuously refreshes questions from recent sources and grades automatically against objective ground truth across many task categories.
BigCodeBench
A code-generation benchmark for realistic programming tasks that require composing many library calls, evaluated with rigorous test suites and high branch coverage.
CRUXEval (Code Reasoning, Understanding, and Execution)
A benchmark that tests whether models can reason about code execution by predicting function inputs from outputs and outputs from inputs.
RewardBench
A benchmark for reward models and LLM judges that measures how well they prefer better responses over worse ones across chat, reasoning, safety, and refusal cases.
AlpacaEval
An automated LLM-as-judge benchmark that estimates a model's win rate against a reference model on open-ended instructions, with a length-controlled variant to reduce verbosity bias.
Arena-Hard
An automatic benchmark of challenging, real-user-derived prompts graded by an LLM judge, built to align closely with human preference rankings and separate strong models.
ImageNet (ILSVRC Classification)
The foundational large-scale image classification benchmark covering 1,000 object categories, long used to track progress in computer vision and pretraining.
COCO (Object Detection and Segmentation)
A large-scale benchmark for object detection, instance segmentation, and keypoints in complex everyday scenes, evaluated with mean average precision across IoU thresholds.
VQAv2 (Visual Question Answering)
A benchmark that tests whether models can answer open-ended natural-language questions about images, balanced to reduce language-only shortcuts.
MMBench (Multimodal Benchmark)
A systematic multimodal benchmark that evaluates vision-language models across many fine-grained ability dimensions using a robustness-checked multiple-choice protocol.
DocVQA (Document Visual Question Answering)
A benchmark for answering questions about document images, testing OCR, layout understanding, and reasoning over text, tables, and forms.
ChartQA
A benchmark for answering questions about charts and plots that require visual data extraction plus arithmetic and logical reasoning over the extracted values.
LibriSpeech (ASR Word Error Rate)
A widely used benchmark for automatic speech recognition built from read English audiobooks, measured primarily by word error rate on clean and noisy splits.
HarmBench
A standardized red-teaming benchmark that measures how often automated attacks elicit harmful behaviors from LLMs and how well refusal and defenses hold up.
MLPerf Training
Industry-standard benchmark suite measuring how fast hardware and software systems train machine-learning models to a fixed target quality.
MLPerf Inference
Benchmark suite measuring how fast and efficiently systems serve trained ML models under realistic latency and throughput constraints.
MLPerf Tiny
Benchmark suite for ultra-low-power machine learning on microcontrollers and embedded devices, measuring latency, energy, and accuracy.
DAWNBench
Stanford benchmark that measured end-to-end deep-learning training and inference by time-to-accuracy and cost, popularizing those metrics.
SPECjbb 2015
Standard Java server benchmark modeling a supermarket company's business logic to measure throughput and critical response-time performance.
SPECpower_ssj2008
Benchmark measuring server energy efficiency by reporting performance per watt across graduated load levels from idle to peak.
SPECjvm 2008
Benchmark suite measuring core Java Virtual Machine performance across compute-intensive workloads independent of application or hardware tuning.
SPEC OMP 2012
Benchmark suite measuring shared-memory parallel performance of OpenMP applications across scientific and engineering workloads.
HPCG
High Performance Conjugate Gradients benchmark measuring HPC system performance on memory-bound, sparse computations that mirror real applications.
HPL-AI / HPL-MxP
Mixed-precision LINPACK variant measuring supercomputer performance using low-precision arithmetic refined to full accuracy, reflecting AI hardware.
Graph500
Benchmark ranking supercomputers on data-intensive graph processing, measuring traversed edges per second instead of floating-point throughput.
GAP Benchmark Suite
Reference graph-algorithm benchmark suite providing optimized kernels and standard graphs to fairly compare graph-processing performance.
Computer Language Benchmarks Game
Long-running comparison of programming-language implementations on small algorithmic tasks, measuring runtime, memory, and code size.
Renaissance JVM Benchmark Suite
Modern JVM benchmark suite using real-world concurrent and parallel workloads to stress runtime optimization, GC, and JIT compilers.
DaCapo JVM Benchmark Suite
Long-established Java benchmark suite using real open-source application workloads to evaluate JVM, JIT, and garbage-collection performance.
SPECviewperf
Standard benchmark measuring professional graphics-workstation performance running real CAD, visualization, and content-creation application viewsets.
3DMark
Cross-platform graphics benchmark suite measuring GPU and gaming performance through standardized rendering tests and synthetic feature tests.
Unigine Superposition
GPU stress and benchmark tool using a detailed real-time scene to measure graphics performance, stability, and thermal behavior.
PassMark PerformanceTest
Whole-system PC benchmark suite measuring CPU, memory, disk, 2D, and 3D performance and aggregating them into comparable component scores.
UnixBench
Classic Unix and Linux system benchmark measuring overall performance through CPU, process, file I/O, and system-call tests aggregated into an index.
sysbench CPU Benchmark
CPU test mode of the sysbench tool measuring processor throughput via prime-number computation across single and multiple threads.
stress-ng
Configurable stress-test and micro-benchmark tool that loads CPU, memory, I/O, and kernel subsystems through hundreds of targeted stressors.