The Fear of Spot
Think of Spot Instances like standby airline seats: dramatically cheaper than a regular ticket, but the airline can bump you if a full-fare passenger shows up. Most teams hear "can be reclaimed at any time" and immediately decide the risk isn't worth it — so they pay full price for every seat, every flight, even when most of those seats are carrying low-priority cargo that nobody would notice if it arrived an hour late.
The result is predictable: 100% On-Demand clusters where a stateless batch job that processes log files costs the same per hour as the production API serving real customers. That's not safety — it's waste dressed up as caution.
The Hybrid Solution
You don't have to choose between "Cheap and Risky" (100% Spot) or "Expensive and Safe" (100% On-Demand). Spot Orchestration is the practice of mixing both capacity types in the same cluster, routing each workload to the tier that matches its actual risk tolerance.
The mental model is simple:
| Workload | Characteristics | Target Capacity |
|---|---|---|
| Production API | User-facing, latency-sensitive | On-Demand |
| Database | Stateful, hard to recover | On-Demand |
| Batch jobs | Retryable, delay-tolerant | Spot |
| CI/CD runners | Ephemeral, idempotent | Spot |
| ML training | Long-running but checkpointed | Spot with fallback |
Karpenter (an open-source node provisioner for Kubernetes, originally developed at AWS) makes this seamless. Unlike the legacy Cluster Autoscaler — which manages pre-defined Auto Scaling Groups (ASGs) and reacts slowly to demand — Karpenter watches the Kubernetes scheduler queue directly and makes provisioning decisions in seconds, choosing the right type of node for the specific pods waiting to be scheduled.
Interviewers frequently ask: "What's the difference between Karpenter and Cluster Autoscaler?" The key distinction is where each tool operates. Cluster Autoscaler scales pre-defined ASGs up or down — it can only choose from node types you configured in advance. Karpenter reads pod requirements directly and calls the cloud provider's Fleet API to launch exactly the right instance type on demand, with no pre-configuration required. This makes Karpenter dramatically faster (seconds vs. minutes) and more cost-efficient.