Spot Orchestration for Stateful Workloads

The "Spot Is Only for Stateless" Myth

Spot instances offer discounts of 60–90% compared to On-Demand pricing. The trade-off is that AWS can reclaim them with a 2-minute warning when capacity is needed elsewhere. Because of this interruption risk, many teams assume Spot is only safe for stateless, disposable workloads — web servers, CI/CD runners, embarrassingly parallel batch jobs.

This assumption is outdated. With the right architecture, you can run stateful workloads on Spot instances and capture most of the savings. The key is designing for interruption tolerance rather than trying to prevent interruptions.

Pattern 1: Checkpointing for Long-Running Jobs

This pattern applies to batch workloads that run for hours — machine learning training, video transcoding, large-scale data processing.

The problem without checkpointing: A model training job runs for 8 hours. At hour 7, the Spot instance is reclaimed. The job restarts from zero on a new instance. You've wasted 7 hours of compute.

The solution: The application periodically saves its current state (a "checkpoint") to durable storage — typically S3 or Amazon EFS. When a new instance starts, it checks for an existing checkpoint and resumes from there.

Job starts → Checkpoint every 10 min to S3
                    ↓
         Spot interruption at T+7h
                    ↓
         New Spot instance starts
                    ↓
         Reads checkpoint from S3 (T+6h50m state)
                    ↓
         Resumes — loses only 10 minutes of work

Many ML frameworks support this natively. PyTorch, for example, can save model state dicts to disk at configurable intervals.

Pattern 2: Decoupled Storage (The Re-attach Strategy)

This pattern applies to stateful services like databases, message brokers, or any application where data must survive instance termination.

The core principle: Separate the compute (the instance) from the data (the EBS volume). The instance is ephemeral; the data is permanent.

The orchestration flow when a Spot interruption occurs:

1. The instance receives the 2-minute termination warning via the EC2 Instance Metadata Service (IMDS).
2. A monitoring script (or Kubernetes operator) detects the warning and triggers a graceful shutdown sequence.
3. The application flushes in-memory state to the EBS volume.
4. The EBS volume is detached from the terminating instance.
5. The instance terminates.
6. An Auto Scaling Group or orchestration layer launches a new Spot instance.
7. The new instance re-attaches the EBS volume and mounts it.
8. The application starts and reads its state from the volume — resuming as if nothing happened.

Total downtime: Typically 2–4 minutes during the swap. For many workloads (development databases, analytics pipelines, non-critical services), this is acceptable in exchange for 70–90% cost savings.

The 2-minute warning detection: A lightweight daemon polls the EC2 instance metadata service's termination-time endpoint every 5 seconds — it returns a 404 when no interruption is scheduled, and a timestamp once one is imminent. That polling cadence gives roughly 24 checks before termination — enough time to flush and detach cleanly.