The Problem: Models Don't Fit
Every number in a neural network is stored as a floating-point value. Standard training uses 32-bit floating-point numbers (FP32), where each parameter occupies 4 bytes of memory. A 70B parameter model at FP32 would need roughly 280 GB of VRAM — more than three top-tier A100 GPUs combined. Even the industry-standard 16-bit format (FP16 or BF16, 2 bytes per parameter) requires ~140 GB, which still exceeds a single GPU's capacity.
This is the VRAM bottleneck: before the model can process a single token, every weight must be resident in GPU memory. You can't page weights in and out the way a CPU can page RAM to disk — the latency would make the model unusable.
The Solution: Quantization
Quantization is the process of reducing the numerical precision of model weights to shrink their memory footprint, ideally without meaningfully degrading output quality.
Think of it like compressing a high-resolution photograph. A raw 48-megapixel image takes enormous storage. A well-compressed JPEG at 90% quality looks nearly identical but is a fraction of the size. Quantization applies the same principle to model weights:
| Format | Bits per weight | Memory for 70B model | Relative quality loss |
|---|---|---|---|
| FP32 | 32 | ~280 GB | None (baseline) |
| FP16/BF16 | 16 | ~140 GB | Negligible |
| INT8 | 8 | ~70 GB | Very small |
| INT4 | 4 | ~35 GB | Small to moderate |
With INT4 quantization, a 70B model can fit on a single high-end GPU (80 GB VRAM) with room to spare for the context cache.
Smart Compression: GPTQ and AWQ
Naive quantization — simply rounding every weight to the nearest integer — causes significant accuracy degradation. The reason is that weights are not uniformly important. Some weights are highly sensitive to rounding errors; others can be aggressively compressed with almost no effect on output.
GPTQ (Generative Pre-trained Transformer Quantization) and AWQ (Activation-aware Weight Quantization) are algorithms that solve this problem. They analyze the model's behavior on a small calibration dataset to identify which weights are sensitive, then apply precision selectively — keeping critical weights more precise while aggressively compressing less important ones.
The practical result: AWQ INT4 quantization of Llama-3-70B typically produces outputs that are nearly indistinguishable from the FP16 version on most benchmarks, while using one-quarter of the VRAM.
Interviewers often ask: "What's the trade-off between INT4 and INT8 quantization?" The answer is not just memory vs. quality — it's also throughput. INT4 models can fit on fewer GPUs, reducing hardware costs and inter-GPU communication overhead. However, INT4 can introduce subtle accuracy regressions on tasks requiring precise numerical reasoning (e.g., math, code). A strong answer acknowledges that the right quantization level depends on the task, not just the hardware budget.