The Hidden Memory Hog: KV Cache
Even after quantizing the model weights, there is a second memory problem: the KV Cache (Key-Value Cache).
When an LLM generates text, it uses an attention mechanism that requires comparing the current token against every previous token in the conversation. To avoid recomputing these comparisons from scratch on every generation step, the model stores intermediate results — called keys and values — in a cache. This is the KV Cache.
The problem: the KV Cache grows with every token generated. A 70B model with a 128k token context window can require tens of gigabytes of VRAM just for the cache of a single long conversation. In traditional serving frameworks, this memory must be reserved upfront for each request — because you don't know how long the conversation will be. If you reserve 20 GB per user and most conversations use only 2 GB, you've wasted 90% of your available memory.
PagedAttention: Virtual Memory for LLMs
vLLM is an open-source, high-performance LLM serving library. Its core innovation is PagedAttention, which borrows a concept from operating system design: virtual memory paging.
In an OS, physical RAM is divided into fixed-size pages. A process doesn't need a contiguous block of RAM — the OS maps logical addresses to physical pages scattered across memory. This eliminates fragmentation and allows memory to be shared efficiently.
PagedAttention applies the same idea to the KV Cache:
- The KV Cache is divided into fixed-size blocks (pages).
- Each page holds the key-value data for a fixed number of tokens.
- Pages are allocated on demand as the conversation grows, and freed immediately when the conversation ends.
- Multiple requests can share pages when they have identical prefixes (e.g., a shared system prompt).
Without vLLM: A parking lot where every car must reserve 10 spaces upfront in case it's a stretch limo. Most spaces sit empty.
With vLLM (PagedAttention): Cars park in single spaces. If a car grows into a limo, the system finds additional empty spaces anywhere in the lot.
The practical impact is dramatic. vLLM benchmarks consistently show 2–4x higher throughput (requests per second) compared to naive serving implementations on the same hardware, because far more concurrent requests can be held in memory simultaneously.
Continuous Batching
vLLM also implements continuous batching (also called iteration-level scheduling). Traditional batch inference waits for a full batch of requests to arrive before processing them together. Continuous batching processes requests at the token level — as soon as one request finishes generating a token, the GPU slot is immediately filled with a new request, rather than waiting for the entire batch to complete. This keeps GPU utilization near 100% even with variable-length requests.