deployment
Quantization
A technique for reducing model size and inference cost by using lower-precision number formats for weights and activations. Quantization can shrink a model from 16-bit to 4-bit precision with minimal quality loss, enabling deployment on consumer hardware.
In practice
A 70B parameter model quantized to 4-bit can run on a single consumer GPU instead of requiring a server cluster.