Inference
17 items tagged with "inference"
Comparisons5
vLLM vs TGI
vLLM and Hugging Face Text Generation Inference (TGI) are high-throughput LLM serving engines. Both optimize GPU inference but differ in ecosystem and tuning focus.
Batch vs Real-Time Inference
Batch inference processes data in scheduled bulk jobs; real-time inference serves predictions on demand. They trade latency against throughput, cost, and complexity.
CPU vs GPU Inference
CPUs and GPUs both run ML inference. GPUs excel at parallel, large-model workloads; CPUs are cheaper and simpler for small models and low concurrency.
Quantization vs Full Precision
Quantization stores model weights at lower bit-widths to cut memory and speed inference; full precision preserves maximum accuracy. The trade-off is size and speed versus fidelity.
Self-Hosted vs Managed Inference
Self-hosted inference runs models on your own infrastructure; managed inference uses a provider's endpoint. The choice trades control, privacy, and cost-at-scale against simplicity.
Benchmarks2
MLPerf Inference
Benchmark suite measuring how fast and efficiently systems serve trained ML models under realistic latency and throughput constraints.
DAWNBench
Stanford benchmark that measured end-to-end deep-learning training and inference by time-to-accuracy and cost, popularizing those metrics.
FAQs3
What is temperature in LLM generation?
Temperature is a parameter that controls the randomness of a language model's output by scaling the probability distribution over the next token. A lo...
What is inference in machine learning?
Inference is the phase where a trained model is used to make predictions or generate output on new inputs, as opposed to training where the model lear...
What is quantization in machine learning?
Quantization reduces the numerical precision of a model's weights and activations, for example from 32-bit floating point to 8-bit or 4-bit integers, ...
Glossaries5
Prompt Engineering
Prompt engineering is the practice of designing and refining the text instructions given to a language model to steer its output toward accurate, useful results.
Inference
Inference is the process of running a trained model on new inputs to produce outputs, as opposed to the training phase that creates the model.
Temperature (LLM Sampling)
Temperature is a sampling parameter that scales an LLM's output probabilities, controlling how random or deterministic its token choices are.
Top-p Sampling (Nucleus Sampling)
Top-p sampling restricts an LLM's next-token choice to the smallest set of tokens whose cumulative probability exceeds a threshold p, then samples from that set.
Quantization
Quantization reduces the numeric precision of a model's weights and activations, shrinking memory use and speeding inference with limited accuracy loss.