The Storage Cost Dilemma
Kafka's default storage model ties data retention directly to broker disk capacity. Every byte you want to retain must live on an EBS volume attached to a broker EC2 instance. This creates a painful coupling: to store more data, you must add more brokers, even if your throughput doesn't require it.
Consider a concrete scenario: A fintech company needs to retain all transaction events for 7 years for regulatory compliance. Their throughput is modest — 5 MB/sec. Three kafka.m5.large brokers handle the throughput easily. But 7 years of data at 5 MB/sec (with 3x replication) is roughly 30 petabytes. Storing that on EBS would cost hundreds of thousands of dollars per month and require hundreds of brokers that sit idle from a CPU perspective.
How Tiered Storage Works
MSK Tiered Storage decouples compute (brokers) from storage (S3) by introducing two storage tiers:
Tier 1 — Local (Hot): Recent data lives on the EBS volumes attached to your brokers. You configure a local retention window (e.g., 2 days). Reads from this tier are low-latency because the data is on local SSD.
Tier 2 — Remote (Cold): Data older than your local retention window is automatically offloaded to Amazon S3 by MSK. The broker handles this transparently — no application code changes required.
Transparent Reads: When a consumer requests data that has been offloaded to S3 (e.g., replaying events from 6 months ago), the broker fetches the data from S3 and serves it to the consumer. The consumer uses the same Kafka consumer API — it has no knowledge of which tier the data came from. Latency for cold reads is higher (S3 fetch overhead), but for compliance replay scenarios this is acceptable.
Cost model: You pay for 3 brokers (sized for throughput, not retention) plus S3 storage at roughly $0.023/GB/month. The 30 petabyte example above becomes economically viable.
Configuration: Tiered Storage is enabled at the cluster level and configured per-topic with two retention settings:
- retention.ms — how long data is retained in total (across both tiers)
- local.retention.ms — how long data stays in the local (hot) tier before offloading to S3
# Topic configuration for tiered storage
retention.ms=157680000000 # 5 years total retention
local.retention.ms=172800000 # 2 days on local EBS (hot tier)
remote.storage.enable=true # Enable tiered storage for this topic
Architectural implication: Tiered Storage changes the broker sizing conversation. Previously, you sized brokers for max(throughput, storage). With Tiered Storage, you size brokers for throughput only. Storage scales independently and cheaply in S3. This is the same architectural shift that columnar data warehouses made when they moved to S3-backed storage (Redshift RA3, Snowflake, Athena).
A common interview question: "How does MSK Tiered Storage affect consumer offset management?" The answer: offsets are unaffected. Kafka offsets are sequential integers per partition. Tiered Storage does not change the offset numbering — data offloaded to S3 retains its original offsets. A consumer seeking to offset 0 on a topic with 3 years of history will transparently receive data fetched from S3. The only observable difference is higher fetch latency for cold data.