The Idle Giant
Imagine you buy a massive, industrial-sized pizza oven for your restaurant. It's powerful enough to bake 100 pizzas at once. But right now, a customer only wants one personal-sized pizza. You have to fire up the entire massive oven just to bake that one tiny pizza. It's a huge waste of energy and capacity.
This is exactly what happens when you give a massive GPU like an NVIDIA A100 to a data scientist working on a small exploratory model, or use it to serve a low-traffic inference endpoint. The GPU spends most of its time waiting, utilizing only a fraction of its compute cores and memory bandwidth. GPU utilization dashboards in these scenarios routinely show 5–15% — meaning you're paying for a sports car and using it to idle in a parking lot.
Enter Multi-Instance GPU (MIG)
NVIDIA's solution to this is Multi-Instance GPU (MIG), introduced with the A100 architecture and continued in the H100.
MIG allows you to partition a single physical GPU into multiple, completely isolated GPU instances. It's like taking that giant pizza oven and installing physical dividers to create up to seven separate, smaller ovens that can all run simultaneously and independently.
Each MIG instance has its own dedicated:
- Streaming Multiprocessors (SMs): The actual compute cores that execute GPU kernels.
- High-Bandwidth Memory (HBM): The GPU's own ultra-fast RAM, partitioned so each slice gets a guaranteed allocation.
- L2 Cache and Memory Bandwidth: Ensuring one workload can't monopolize the memory bus and starve others.
Because the hardware is physically partitioned at the silicon level — not just virtualized in software — work running on one slice genuinely cannot impact the performance of work on another slice. This fault isolation is crucial for multi-tenant environments where several different teams or applications share the same physical hardware. You can run a small training job on one slice and a production inference server on another, simultaneously, on one physical card, with guaranteed performance for each.
MIG Profiles: Choosing Your Slice Size
NVIDIA defines standard partition sizes called MIG profiles. On an A100 80GB, the available profiles include:
| Profile | GPU Fraction | Memory | Max Instances |
|---|---|---|---|
1g.10gb |
1/7th | 10 GB | 7 |
2g.20gb |
2/7th | 20 GB | 3 |
3g.40gb |
3/7th | 40 GB | 2 |
7g.80gb |
Full GPU | 80 GB | 1 |
You mix and match these profiles to fit your workload mix. For example, you might carve one A100 into two 3g.40gb instances for two medium training jobs, plus one 1g.10gb instance for a lightweight inference service — all running simultaneously on one physical card.
Enabling and managing MIG is all done through the nvidia-smi command-line tool: one command flips on MIG mode for a GPU, another creates an instance of a given profile, and a third lists the instances currently active — each newly created instance shows up with its own profile ID and placement slot on the physical die.
Interviewers often ask: "What's the difference between MIG and time-slicing?" Time-slicing (also called GPU sharing via the NVIDIA device plugin) allows multiple processes to share a GPU by rapidly switching between them — similar to CPU context switching. MIG provides spatial partitioning with hardware-level isolation and guaranteed resources. Time-slicing has no memory isolation (one process can crash another) and no performance guarantees. For production multi-tenant workloads, MIG is the correct answer. Time-slicing is only appropriate when you need more concurrent processes than MIG instances allow and can tolerate the lack of isolation.
Key Point: MIG is not universally applicable. It requires specific GPU models (A100, H100, A30) and specific CUDA workload patterns. Some operations — particularly those that rely on the full GPU's NVLink bandwidth for multi-GPU communication — cannot be split across MIG instances. Always validate that your specific model and framework support MIG before committing to it in production.