Model quantization means replacing the high-precision numbers used for a model's parameters with lower-precision numbers — for example, going from 32-bit floating point to 8-bit integers or 8-bit floating point — so the model becomes smaller and uses less VRAM and memory, allowing phones and ordinary laptops to run large models that would otherwise not fit. It does not change the model's structure, only how its numbers are represented; it is an approximation that trades a little precision for speed and size.
What quantization actually compresses
Quantization mainly compresses weights — the parameters saved after training. The 32-bit floating-point numbers commonly used in training take 4 bytes per parameter; switching to 8-bit integers takes only 1 byte, shrinking the size to about a quarter of the original. Activations can also be quantized, but they are more sensitive, and compressing them too aggressively can amplify errors early in the intermediate steps of inference. There are two common approaches: post-training quantization, which converts a trained model directly and is convenient but makes precision loss hard to control; and quantization-aware training, which lets the model adapt to low precision during training or fine-tuning, giving more stable results at the cost of an extra training round. Some implementations also keep particularly sensitive layers at high precision and compress only the less sensitive parts.
Will it make the model dumber, and how can you tell?
Whether it becomes dumber depends on the task and how far it is compressed. 8-bit quantization usually has very little effect on everyday chat, summarization, and classification, and many people notice no difference in use; at 4 bits and below, mathematical reasoning, long code, and multi-step reasoning are more likely to break first. Judging it requires looking at more than size: how much the measured scores on the same task differ between the quantized version and the original, and whether your own real inputs show obvious degradation. The same model often has several quantized versions; when choosing, look first at the quantization method and bit width, then at evaluations and user feedback — don't just pick the smallest file.
When not to use a quantized version directly
If VRAM and memory are already sufficient and results need to be as accurate as possible, use the original first. When precise numerical computation, multi-step reasoning, or high-risk content is involved, don't use a quantized version as the final answer either. Quantized versions are best suited to device-constrained, offline, latency- and cost-sensitive scenarios where results are reviewed by someone or errors can be tolerated. Remember one boundary: quantization is not pruning, and it is not distillation. It does not remove parameters or swap in a smaller model; it only records numbers more cheaply. It cannot fix a model's tendency to get things wrong on its own.
