Two Kinds of Work
Not all AI workloads stress the hardware the same way. Engineers classify workloads into two categories based on what limits their speed:
-
Compute-bound: The bottleneck is raw arithmetic. The GPU is doing so many multiplications and additions that the compute cores are the limiting factor. Memory can keep up. Example: training a convolutional neural network (CNN) on image batches, where the same weights are reused many times per batch.
-
Memory-bound: The bottleneck is data movement. The GPU reads a large amount of data, performs relatively little math on it, and writes results back. The compute cores are fast enough — they're just waiting for data. Example: LLM inference (text generation).
Why LLM Inference Is Memory-Bound
Generating text with a large language model is one of the most memory-bandwidth-intensive workloads in existence. Here is why:
To generate one single token (roughly one word), the GPU must:
1. Load every single weight of the active model from HBM into the compute cores.
2. Perform a matrix multiplication.
3. Produce one output vector.
4. Repeat for the next token.
For a 70B parameter model stored in float16 (2 bytes per parameter):
- Model size: 70 × 10⁹ × 2 bytes = 140 GB
- HBM bandwidth (H100): 3.35 TB/s
- Theoretical max throughput: 3,350 GB/s ÷ 140 GB/token ≈ ~24 tokens/second
This ceiling is set entirely by memory bandwidth, not compute speed. Buying a GPU with twice the FLOPS but the same HBM bandwidth would not make inference faster. This is the memory wall for LLMs.
Interviewers at AI infrastructure companies frequently ask: "Why can't we just use a faster GPU to speed up LLM inference?" The correct answer is that LLM inference is memory-bound, not compute-bound. The bottleneck is HBM bandwidth — how fast weights can be streamed from memory to compute cores. Techniques like quantization (reducing weight precision from float16 to int8 or int4) directly address this by shrinking the data that must be read per token, effectively doubling or quadrupling the token throughput without changing the hardware.
Practical Implication: Quantization
Because inference is memory-bound, reducing the size of the model weights directly improves throughput:
| Precision | Bytes/Param | 70B Model Size | Theoretical Tokens/s (H100) |
|---|---|---|---|
| float32 | 4 | 280 GB | ~12 |
| float16 | 2 | 140 GB | ~24 |
| int8 | 1 | 70 GB | ~48 |
| int4 | 0.5 | 35 GB | ~96 |
This is why quantization is not just a memory-saving trick — it is a throughput optimization that directly exploits the memory-bound nature of LLM inference.