Enhanced Fan-Out: Breaking the Read Limit

The Polling Problem

By default, Kinesis consumers use the GetRecords API to poll a shard — they periodically ask "do you have new data for me?" This is called the standard iterator model.

The problem: a single shard supports only 5 GetRecords API calls per second and a total of 2 MB/sec of read throughput shared across all consumers. If three separate applications (a Lambda function, an EC2 analytics job, and a monitoring agent) all poll the same shard, they share that 2 MB/sec budget. Each gets roughly 667 KB/sec. As you add more consumers, each gets less throughput and higher latency.

The Solution: Enhanced Fan-Out (EFO)

Enhanced Fan-Out (EFO) replaces polling with HTTP/2 server push. Instead of consumers asking for data, Kinesis pushes data to each registered consumer the moment it arrives.

How it works:
1. You register a consumer with RegisterStreamConsumer.
2. The consumer calls SubscribeToShard to open a persistent HTTP/2 connection.
3. Kinesis pushes records to the consumer as they arrive — no polling loop needed.

The key difference: Each EFO consumer gets its own dedicated 2 MB/sec pipe per shard, independent of all other consumers. Three consumers on the same shard each get 2 MB/sec — 6 MB/sec of total read throughput from a shard that nominally supports 2 MB/sec for standard consumers.

When to use EFO: When you have 2 or more consumers that need low latency and cannot afford to share throughput. The extra cost is justified when consumer lag or throttling would cause business impact.