The Illusion of One Memory
Before CUDA 6 (2014), GPU programming required explicit memory management. You allocated memory on the CPU with malloc, allocated separate memory on the GPU with cudaMalloc, and manually copied data between them with cudaMemcpy. Forgetting a copy was a common source of bugs — the GPU would silently read stale or zeroed data.
Unified Memory (also called Managed Memory) presents a single virtual address space shared between the CPU and GPU. You allocate it once:
float *data;
cudaMallocManaged(&data, size);
// Now 'data' is valid on both CPU and GPU
The pointer works on both sides. The driver handles movement automatically.
Under the Hood — GPU Page Faults:
Unified Memory works through the virtual memory system. When you first allocate managed memory, no physical pages are committed anywhere. When the GPU tries to access a page that currently lives in CPU RAM (or hasn't been allocated yet), the GPU's Memory Management Unit (MMU) raises a GPU Page Fault — the GPU equivalent of a CPU page fault.
The sequence:
1. GPU thread accesses address 0xABCD....
2. GPU MMU finds no valid mapping → raises a page fault interrupt.
3. The kernel driver's fault handler wakes up on the CPU.
4. The handler copies the relevant memory page from CPU RAM to GPU VRAM over PCIe.
5. The GPU page table is updated with the new physical address.
6. The faulting GPU thread resumes.
This "lazy migration" is convenient but has a serious performance failure mode: page thrashing. If your CPU preprocesses a batch and writes to managed memory, then the GPU reads it, then the CPU writes the next batch to the same pages before the GPU is done — the pages bounce back and forth across PCIe continuously. Each round trip is ~10–50 µs. At scale, this destroys throughput.
Mitigation strategies:
- Use cudaMemPrefetchAsync() to proactively migrate pages before they are needed, hiding latency behind computation.
- Use cudaMemAdvise() with hints like cudaMemAdviseSetReadMostly (replicate the page on both CPU and GPU) or cudaMemAdviseSetPreferredLocation (keep the page on a specific device).
- For production training pipelines, explicit cudaMemcpy with double-buffering is often faster than Unified Memory because it gives you precise control over when transfers happen.