Think of a model's weights like measurements in a recipe. A professional chef might measure ingredients to six decimal places using a precision scale — but a home cook gets nearly identical results measuring to the nearest gram. Quantization is the engineering equivalent of that trade-off: deliberately using less precise numbers to make a model faster and smaller, while keeping the results good enough to be useful.
Why Are Models So Big?
When you train a neural network in PyTorch or TensorFlow, every weight and activation is stored as FP32 — a 32-bit floating-point number. FP32 is the default because it can represent an enormous range of values with high precision (e.g., 0.00001234567), which is critical during training when tiny gradient updates must accumulate correctly over millions of steps.
The cost is significant:
- Memory: Every single number consumes 4 bytes of RAM. A model with 7 billion parameters (like a small LLM) requires roughly 28 GB just to store the weights — before any inference overhead.
- Compute: Floating-point multiplication is expensive. Modern hardware can perform far more integer operations per second than floating-point operations.
- Bandwidth: GPUs and CPUs must continuously stream weights from memory into compute units. The more data to move, the slower the system — regardless of raw compute power.
The Quantization Solution
Quantization maps high-precision floating-point numbers to lower-precision integers — most commonly INT8 (8-bit signed integers, ranging from -128 to 127).
| Format | Bits | Bytes per value | Relative size |
|---|---|---|---|
| FP32 | 32 | 4 | 1× (baseline) |
| FP16 | 16 | 2 | 2× smaller |
| INT8 | 8 | 1 | 4× smaller |
| INT4 | 4 | 0.5 | 8× smaller |
The benefits of INT8 quantization are concrete:
- 4× smaller model — a 28 GB model becomes ~7 GB.
- Faster inference — integer arithmetic is simpler on silicon; hardware can execute more operations per clock cycle.
- Lower cost — smaller models fit on cheaper hardware, reducing the cost per inference.
The fundamental trade-off is precision loss. The real number 0.00001234 might round to 0 in INT8. The engineering challenge is to lose precision without meaningfully degrading the model's output quality.