Sparse Indexes: The "To-Do List" Pattern

The Cost of Indexing Everything

A Global Secondary Index (GSI) in DynamoDB normally mirrors every item from the main table into the index. If your table has 10 million items, your GSI also has 10 million items — doubling your storage cost and adding a write to the GSI for every write to the main table.

But what if you only care about a tiny subset of items? Indexing everything to find a few items is like building a full library catalog just to track which books are currently checked out.

How Sparse Indexes Work

DynamoDB has a specific behavior: an item is only copied to a GSI if the item contains the attribute(s) used as the GSI's key. If the GSI key attribute is absent from an item, that item is silently excluded from the index.

This is not a configuration option — it is the default behavior. You exploit it deliberately by only writing the GSI key attribute to items that belong in the index, and removing it from items that should leave the index.

The Workflow Pattern: Open Orders

Imagine an Orders table with 10 million orders:

• 9,999,000 are COMPLETED.
• 1,000 are OPEN (need processing).

You want a dashboard showing only open orders.

Bad approach: Scan the entire table with a filter expression for Status = "OPEN". This reads 10 million items and discards 9,999,000 of them. You pay for every read.

Sparse index approach:

1. Create a GSI with Partition Key: IsOpen.
2. When an order is created, write IsOpen = "YES" to the item. DynamoDB copies it to the GSI.
3. When an order is completed, use UpdateItem to delete the IsOpen attribute (REMOVE IsOpen). DynamoDB removes it from the GSI automatically.

The GSI now contains only the 1,000 open orders. A full Scan of this GSI reads 1,000 items instead of 10 million — a 10,000× reduction in cost and latency.

Other Use Cases for Sparse Indexes

• Unprocessed jobs: Write a PendingAt timestamp when a job is queued; remove it when the job completes. The GSI contains only pending jobs, sortable by queue time.
• Flagged content: Write FlaggedAt when a post is reported; remove it after moderation. The GSI is your moderation queue.
• Expiring sessions: Write ExpiresAt for active sessions; remove it on logout. The GSI contains only live sessions.

The pattern is always the same: the GSI key attribute is a lifecycle flag that exists only during the phase you want to query.