What is quantization in machine learning?

Answer

Quantization reduces the numerical precision of a model's weights and activations, for example from 32-bit floating point to 8-bit or 4-bit integers, to shrink memory use and speed up inference. It lets large models run on smaller or cheaper hardware with only modest accuracy loss when done carefully. Post-training quantization applies after training, while quantization-aware training accounts for the lower precision during training to preserve accuracy.