Inference

Inference is the stage where a trained model is used to make predictions or generate outputs from new, unseen inputs. It is distinct from training, which adjusts the model's weights; during inference the weights are fixed and the model simply computes a result.

How It Works

For a language model, inference runs the input tokens through the network to produce a probability distribution over the next token, samples or selects a token, and repeats until a stop condition. Performance is often described with two phases: prefill, which processes the prompt, and decode, which generates output one token at a time. Techniques such as key-value caching, batching, and quantization speed up inference and lower memory use. Inference runs on CPUs, GPUs, or specialized accelerators.

Why It Matters

Inference is where most production cost and latency live, because a deployed model serves many requests over its lifetime. Optimizing throughput and latency, choosing the right hardware, and managing context length are central to running models affordably at scale. Decisions like model size, quantization, and batching trade off speed, cost, and quality.

Related Terms

Inference follows training, generates token sequences from an large-language-model, is bounded by the context-window, and is accelerated by quantization.