What Is a Shard?
In SQS, you send messages and AWS handles the underlying capacity invisibly. In Kinesis, you must explicitly provision capacity using shards. A shard is the fundamental unit of throughput in a Kinesis stream — think of it as a single lane on a highway.
Each shard provides exactly:
- Write: 1 MB/sec OR 1,000 records/sec (whichever limit is hit first).
- Read: 2 MB/sec.
These limits are hard. Exceeding them produces a ProvisionedThroughputExceededException.
Capacity Planning
Scenario: You need to ingest 5 MB/sec of data.
Required shards = Ingestion rate / Per-shard write capacity
Required shards = 5 MB/sec ÷ 1 MB/shard = 5 shards
You provision 5 shards. Total stream capacity: 5 MB/sec write, 10 MB/sec read.
Partition Keys and Shard Routing
Every record you write to Kinesis must include a Partition Key — a string you choose (e.g., UserID, DeviceID, CountryCode). Kinesis applies MD5 hashing to this key and maps the result to a specific shard. Records with the same partition key always land on the same shard, which guarantees ordering for that key.
Partition Key → MD5 Hash → Shard Assignment
"user-42" → 0x3A... → Shard 2
"user-99" → 0xF1... → Shard 4
"user-42" → 0x3A... → Shard 2 (always the same shard)
The Hot Shard Problem
If your partition key has low cardinality (few unique values) or skewed distribution (one value dominates), you get a hot shard: one shard receives the majority of traffic while others sit idle.
Example: You use CountryCode as the partition key. 90% of traffic is from US. The US key hashes to Shard 1. Shard 1 receives 0.9 × 5 MB/sec = 4.5 MB/sec — well above its 1 MB/sec limit. You get throttling errors on Shard 1 while Shards 2–5 are nearly empty.
Fix: Use a high-cardinality partition key that distributes load evenly. Instead of CountryCode, use CountryCode + UserID or CountryCode + random_suffix. This spreads traffic across all shards.
Resharding: Changing Capacity
To change stream capacity, you split or merge shards via API:
- Split a shard: Divide one shard into two. Increases total capacity. Use when a shard is hot or you need more throughput.
- Merge two shards: Combine two adjacent shards into one. Decreases cost. Use when shards are underutilized.
Resharding is not instantaneous — it takes a few seconds and the old shard enters a CLOSED state (still readable for existing data, but no new writes). This is a manual or scripted operation unless you use On-Demand mode, which automatically scales shard count based on observed throughput (at a higher per-GB cost).
Interviewers frequently ask: "How does Kinesis guarantee ordering?" The answer is: ordering is guaranteed per shard, not per stream. Records with the same partition key always go to the same shard and are read in sequence. If you need global ordering across all records, you need a single shard (which limits you to 1 MB/sec). This is a deliberate trade-off between ordering guarantees and throughput.