Mixed-Precision Quantization: Not All Layers Are Equal

The Sensitivity Problem

Quantizing every layer uniformly to INT8 is the simplest approach, but it's not always optimal. Some layers are far more sensitive to quantization error than others:

• First and last layers — the input layer processes raw data (pixels, tokens) where precision matters; the output layer produces final logits where small errors change predictions. These are typically kept in FP16 or FP32.
• Attention layers in transformers — softmax operations and the attention score computation are sensitive to precision; quantizing them aggressively often causes disproportionate accuracy loss.
• Layers with outlier activations — some layers produce activation values with extreme outliers (e.g., a value of +500 when most values are between -2 and +2). A single outlier forces a large Scale, wasting precision for all other values.

Mixed-Precision Strategy

Mixed-precision quantization assigns different precision levels to different layers based on their sensitivity:

Layer 1 (embedding):     FP16  ← sensitive, keep higher precision
Layers 2–10 (attention): INT8  ← quantized
Layers 11–20 (FFN):      INT8  ← quantized
Layer 21 (output):       FP16  ← sensitive, keep higher precision

Tools like NVIDIA's TensorRT and Hugging Face's optimum library can automatically profile layer sensitivity and assign precision levels. The result is a model that captures most of the size and speed benefits of INT8 while preserving accuracy in the layers that need it most.

Beyond INT8: INT4 and Sub-Byte Quantization

For very large language models where even INT8 is too large to fit in GPU memory, INT4 quantization (4-bit integers) is increasingly used. Techniques like GPTQ and AWQ (Activation-aware Weight Quantization) apply INT4 quantization to weights while keeping activations in FP16, achieving 8× size reduction with acceptable accuracy loss on models with 7B+ parameters. This is what enables running a 70B parameter model on a single consumer GPU.