Data Never Stops
Sometimes, "as of last night" isn't good enough. For fraud detection, stock trading, real-time recommendation engines, or system health monitoring, you need insights milliseconds after the data is generated.
Stream Processing handles data as an unbounded, continuous flow of events. Rather than waiting for a batch to accumulate, it processes each event — or micro-batch of events — as it arrives.
The Backbone: Apache Kafka
Before discussing stream processors, it's important to understand Apache Kafka, because it is the most common source of data for stream processing systems.
Kafka is a distributed, fault-tolerant message broker and event log. Producers write events to named topics. Consumers read from those topics. Kafka retains events for a configurable period (days or weeks), meaning multiple independent consumers can read the same stream at their own pace.
Think of Kafka as a high-throughput, durable conveyor belt. Stream processors like Flink or Kafka Streams sit at the end of that conveyor belt and process items as they arrive.
The Challenge of Infinity
In batch processing, you know when you're done: when you've read the last record of the file. In streaming, you are never "done." This introduces complex problems that don't exist in batch:
| Challenge | Description |
|---|---|
| Windowing | How do you calculate "events per hour" over an infinite stream? |
| Out-of-order data | An event from 10:00 AM may arrive at 10:05 AM due to network lag. |
| State | How do you remember past events to compare with current ones? |
| Exactly-once semantics | If a server crashes mid-processing, how do you avoid double-counting? |
The Champion: Apache Flink
Apache Flink is widely considered the most advanced and robust framework for stateful stream processing. Unlike Spark Streaming (which processes micro-batches — small batches collected over a short interval), Flink was designed from the ground up to process events one at a time with true low latency, typically in the range of milliseconds.