What is inference in machine learning?

Answer

Inference is the phase where a trained model is used to make predictions or generate output on new inputs, as opposed to training where the model learns from data. For LLMs, inference means running a forward pass to produce tokens, and it is where latency, throughput, and serving cost matter most in production. Techniques such as batching, caching, and quantization are used to make inference faster and cheaper at scale.