The Co-Processor Model
Think of your CPU as a general contractor — skilled at managing complex, varied tasks in sequence. A GPU is more like a factory floor with thousands of workers, each doing a simple, identical job simultaneously. When you add a GPU or TPU to a server, you are not upgrading your CPU — you are plugging in a second, specialized computer that operates largely independently, with its own workforce, its own memory, and its own instruction set.
This co-processor model has a fundamental consequence: the two computers must communicate. That communication channel is the PCIe Bus (Peripheral Component Interconnect Express), a physical electrical connection on the motherboard. PCIe is fast by everyday standards — tens of gigabytes per second — but it is orders of magnitude slower than the internal memory bandwidth of either chip. A modern GPU's internal memory (HBM2e) moves data at over 2 TB/s internally. PCIe Gen4 x16 tops out around 32 GB/s in each direction. That is a 60× gap.
The entire discipline of GPU performance engineering is, at its core, about minimizing how often you cross that PCIe bridge. Every architectural decision — Unified Memory, NVLink, GPUDirect RDMA — is a response to this one physical constraint.
Key components in the co-processor model:
- Own Architecture: The GPU has its own cores (CUDA cores, Tensor cores), its own instruction set (PTX is the virtual ISA; SASS is the actual machine code), and its own memory hierarchy (L1/L2 cache → HBM/GDDR VRAM).
- The Bottleneck: CPU and GPU are separated by the PCIe Bus. Data must be explicitly moved across it, or the hardware must be configured to hide that movement.
- The Goal: Keep the GPU fed with work and data without stalling on PCIe transfers.