How Firehose Works
Firehose is a fully managed delivery pipeline. You send records to it, and it handles batching, compression, optional transformation, and delivery to a destination. You write no consumer code.
Firehose uses two buffer triggers to decide when to write a batch:
- Buffer Size: (1 MB – 128 MB). "Accumulate data until I have this much, then write."
- Buffer Interval: (60 – 900 seconds). "Wait this long, then write whatever I have, even if the buffer isn't full."
Whichever condition is met first triggers the write. If you set 5 MB / 60 seconds, and 5 MB arrives in 20 seconds, Firehose writes at 20 seconds. If only 1 MB arrives in 60 seconds, Firehose writes at 60 seconds.
This means the minimum latency for data to appear at the destination is 60 seconds. Firehose is explicitly a near-real-time service, not a real-time one.
Optional Lambda Transformation
Before delivering to the destination, Firehose can invoke a Lambda function to transform each record — for example, to parse JSON, filter fields, or convert formats. The transformed records are then buffered and delivered. If transformation fails, Firehose can write failed records to a separate S3 prefix for debugging.
Destinations
Firehose supports four core destinations:
| Destination | Notes |
|---|---|
| Amazon S3 | The universal catch-all. Files are written as objects with configurable prefixes. |
| Amazon Redshift | Firehose writes to S3 first, then issues a COPY command to load into Redshift. S3 acts as a staging area. |
| Amazon OpenSearch | Streams logs or events for near-real-time search and dashboards. |
| Splunk | Delivers to Splunk HTTP Event Collector (HEC) for operational intelligence. |
The "Real-Time" Myth
A common interview trap: candidates say Firehose is "real-time" because it's fast. It is not. The 60-second minimum buffer interval means data is always at least one minute old by the time it reaches S3. For sub-second or even sub-minute latency requirements, you must use Kinesis Data Streams with a custom consumer (Lambda, KCL application, or Flink).
A common interview question: "Can Kinesis Firehose read from a Kinesis Data Stream?" Yes — Firehose can use a Kinesis Data Stream as its source instead of receiving records directly. This pattern lets you use KDS for real-time processing (e.g., fraud detection Lambda) while simultaneously using Firehose to archive the same stream to S3 for long-term storage. The stream is the source of truth; Firehose is just one of its consumers.