There are two fundamentally different moments at which you can apply quantization: after training is complete, or during training itself. Each approach has a distinct cost-accuracy profile.
1. Post-Training Quantization (PTQ) — "The Afterthought"
PTQ is the simpler path. You train your model normally in FP32, and once training is complete, you convert it to INT8 using calibration.
Process:
1. Train model in FP32 (standard process, no changes).
2. Run calibration with a representative dataset.
3. Convert weights and configure activation quantization parameters.
4. Export the quantized model.
Pros:
- Works on any pre-trained model, including models you didn't train yourself (e.g., downloaded from Hugging Face).
- No changes to the training pipeline.
- Fast to apply — calibration takes minutes, not hours.
Cons:
- Accuracy can drop, sometimes significantly. The model was never exposed to quantization noise during training, so its weights are not optimized to tolerate it.
- Small models suffer more than large models. A large model has redundant capacity that absorbs quantization error; a small model does not.
- Certain layer types (e.g., the first and last layers, attention layers in transformers) are particularly sensitive to quantization and may need to remain in FP32.
When to use PTQ: Large models (7B+ parameters), latency-tolerant applications, or when you don't have access to the training pipeline. PTQ is the default starting point — try it first, and only escalate to QAT if accuracy is unacceptable.
2. Quantization-Aware Training (QAT) — "The Simulation"
QAT integrates quantization into the training loop itself. The model learns to produce good outputs despite the precision constraints it will face at inference time.
Process:
1. Start with a pre-trained FP32 model (or train from scratch).
2. Insert fake quantization nodes into the computation graph — these simulate the rounding that INT8 would apply, but the actual computation still runs in FP32.
3. Continue training (or fine-tune) with the fake quantization nodes active.
4. During the forward pass, activations and weights are rounded to simulate INT8 precision.
5. During the backward pass, gradients flow through as if the rounding didn't happen (using the Straight-Through Estimator — a trick that treats the rounding function as having a gradient of 1).
6. The optimizer updates weights to minimize loss given the quantization noise.
7. After training, convert to actual INT8.
Pros:
- Highest possible accuracy after quantization. The model has explicitly learned to compensate for INT8 limitations.
- Can recover accuracy that PTQ loses, often getting within 0.5% of the original FP32 model.
Cons:
- Requires access to the training pipeline and training data.
- Adds complexity to the training setup.
- Takes longer — you're running additional training epochs.
When to use QAT: Small models where PTQ causes unacceptable accuracy loss, safety-critical applications, or when you're building a model from scratch and know it will be deployed quantized.
A common interview question: "Your team quantized a model with PTQ and accuracy dropped from 94% to 87%. What do you do?" The structured answer: First, check if the calibration dataset is representative — poor calibration is the most common cause of unexpected accuracy drops. Second, try mixed-precision quantization: keep sensitive layers (first layer, last layer, attention heads) in FP16 or FP32 while quantizing the rest. Third, if accuracy is still unacceptable, escalate to QAT. Don't jump straight to QAT — it's expensive and often unnecessary.