Automating AWS with `boto3`: Paginators and Waiters

Handling Cloud Scale and Asynchrony

boto3 is the official AWS SDK for Python. It lets you manage every AWS service programmatically — EC2, S3, RDS, IAM, Lambda, and hundreds more. Two patterns that every ops engineer must understand are paginators (for large datasets) and waiters (for asynchronous operations).

Paginators — When Results Don't Fit in One Response

AWS API responses are paginated. If you have an S3 bucket with 50,000 objects and you call list_objects_v2, AWS returns the first 1,000 objects and a NextContinuationToken. To get the next 1,000, you must make another call with that token. And again. And again.

If you write a naive loop without handling pagination, you silently process only the first 1,000 objects and never know you missed the rest. This is a common source of subtle bugs in ops scripts — the script appears to work, but it is operating on an incomplete view of reality.

Without a paginator (broken for large buckets):

import boto3

s3 = boto3.client('s3')

# This only returns the first 1,000 objects. Silent data loss.
response = s3.list_objects_v2(Bucket='my-large-bucket')
for obj in response.get('Contents', []):
    print(obj['Key'])

With a paginator (correct):

import boto3

s3 = boto3.client('s3')
paginator = s3.get_paginator('list_objects_v2')

# page_iterator automatically fetches all pages, handling tokens for you
page_iterator = paginator.paginate(Bucket='my-large-bucket')

for page in page_iterator:
    for obj in page.get('Contents', []):
        print(obj['Key'])

The paginator handles the token exchange in the background. You iterate over all objects as if they were in a single flat list. This pattern works for any boto3 API that supports pagination — EC2 instances, IAM users, CloudWatch log groups, and more.

Waiters — Blocking Until the Cloud Catches Up

Cloud operations are asynchronous. When you call run_instances to start an EC2 instance, the API returns immediately with the instance ID, but the instance is in a pending state. It takes 30–90 seconds to boot. If your script immediately tries to SSH into it or attach an EBS volume, it will fail.

The naive fix is time.sleep(90) — but this is fragile. Sometimes instances boot in 20 seconds; sometimes they take 3 minutes. A hardcoded sleep either wastes time or causes failures.

Waiters solve this correctly. They poll the AWS API on a configurable interval until a specified condition is met, then unblock your script.

import boto3

ec2 = boto3.client('ec2', region_name='us-east-1')

# Start an instance
response = ec2.run_instances(
    ImageId='ami-0abcdef1234567890',
    InstanceType='t3.micro',
    MinCount=1,
    MaxCount=1
)
instance_id = response['Instances'][0]['InstanceId']
print(f"Instance {instance_id} is starting...")

# Create a waiter for the 'instance_running' state
waiter = ec2.get_waiter('instance_running')

# This call BLOCKS until the instance reaches 'running' state
# boto3 polls every 15 seconds, up to 40 times (10 minutes max)
waiter.wait(InstanceIds=[instance_id])

print(f"Instance {instance_id} is now running. Safe to proceed.")
# Now it is safe to describe the instance, get its public IP, SSH in, etc.

Waiters exist for dozens of AWS states: instance_running, instance_stopped, instance_terminated, db_instance_available, bucket_exists, stack_create_complete, and more. Always prefer a waiter over time.sleep().