The Cloud's Clearance Aisle
Cloud providers like AWS, Google Cloud, and Azure build massive data centers with enough servers to handle peak demand — think Black Friday shopping traffic or a viral product launch. Most of the time, a significant portion of that capacity sits idle.
Rather than let it go to waste, they offer this spare capacity at massive discounts — often 60–90% off the regular "on-demand" price. These are called Spot Instances (AWS), Preemptible VMs (Google Cloud), or Spot VMs (Azure). For GPU instances, which are already expensive, these discounts are transformative. A training job that costs $10,000 on on-demand instances might cost $1,000–$4,000 on spot.
The Catch: You Can Be Evicted
The catch is that the cloud provider can reclaim the server at any moment if demand for on-demand capacity increases. They typically give you a very short warning — usually just 2 minutes on AWS (via the EC2 Instance Metadata Service), 30 seconds on Google Cloud — before terminating your instance.
If your training job has been running for 3 days and gets terminated without saving its progress, you've just wasted 3 days of compute time and the associated cost. The spot discount becomes meaningless if you lose all your work.
When Spot Makes Sense (and When It Doesn't)
Spot instances are ideal for:
- Model training: Long-running jobs that can be checkpointed and resumed.
- Hyperparameter search: Many independent short jobs where losing one doesn't matter.
- Batch inference: Processing large datasets where individual failures can be retried.
Spot instances are not appropriate for:
- Real-time inference serving: You cannot afford random 2-minute termination windows in a production API.
- Stateful workloads without checkpointing: Any job that cannot be safely interrupted and resumed.