The Problem with Activations
For weights (the static parameters stored in the model file), finding the quantization range is straightforward: scan the file, find the min and max values, done.
Activations are different. Activations are the intermediate values that flow through the network during inference — the outputs of each layer before they become inputs to the next. These values are not stored in the model file; they are computed fresh for every input. A layer that processes a photo of a bright sky produces different activation values than the same layer processing a dark indoor scene.
If you guess the activation range incorrectly:
- Too narrow: Values outside the range get clipped to the boundary. A real value of +100.0 gets treated as +20.2. This destroys information and degrades accuracy.
- Too wide: The Scale becomes large, meaning each integer step represents a big jump in real value. Quantization error increases, and precision is wasted.
The Calibration Process
Calibration is a measurement phase that runs before deployment to empirically determine the correct activation ranges:
- Take the trained FP32 model — weights are already known.
- Assemble a Representative Dataset — a small sample (typically 100–1000 examples) that reflects the real distribution of inputs the model will see in production. This is not a training set; it's just used for measurement.
- Run inference — pass the calibration data through the model in FP32 mode.
- Record statistics — at each layer, observe the min and max (or percentile-based range) of activation values.
- Compute Scale and Zero Point — fix these parameters based on the observed statistics.
The resulting Scale and Zero Point values are then baked into the quantized model and used at inference time.
Calibration Strategies
Different strategies exist for determining the range from observed statistics:
| Strategy | Description | Trade-off |
|---|---|---|
| Min/Max | Use the absolute min and max observed | Simple, but outliers inflate the range |
| Percentile | Use e.g. 99.9th percentile, ignoring outliers | More robust; slight clipping of rare extremes |
| KL Divergence | Find the range that minimizes information loss between FP32 and INT8 distributions | Best accuracy, more compute-intensive |
The quality of your calibration dataset directly determines the quality of your quantized model. A calibration set that doesn't represent production inputs will produce a poorly calibrated model that degrades in real use.