Windowing: Chop the Infinite Stream into Finite Pieces
Since you can't calculate an average or a count over infinite data, you must chop the stream into finite, bounded chunks called Windows. Flink supports several window types:
- Tumbling Window: Non-overlapping, fixed-size time intervals. Example: "Count events every 5 minutes." Windows are: 10:00–10:05, 10:05–10:10, 10:10–10:15. Each event belongs to exactly one window.
- Sliding Window: Overlapping time intervals. Example: "Count events in the last 5 minutes, recalculated every 1 minute." Windows overlap, so an event at 10:03 belongs to the windows starting at 10:00, 10:01, 10:02, and 10:03. Useful for rolling averages.
- Session Window: Groups events that are close together in time, with a gap of inactivity closing the window. Useful for user session analysis where you don't know how long a session will last.
State Management: The Memory of the Stream
If you just want to filter a stream (e.g., "drop all log events with level=INFO"), you don't need memory. Each event is evaluated independently. But if you want to count events in a window, detect a sequence of events, or join two streams, you need to "remember" information across events. This memory is called State.
Flink manages state as a first-class citizen:
- State is stored in fast in-memory structures (by default, a hash map per operator).
- Flink periodically takes consistent checkpoints — atomic snapshots of all operator state — and writes them to durable storage (like S3 or HDFS).
- If a Flink task manager crashes, the job restarts from the last checkpoint. Because the checkpoint is consistent (all operators are snapshotted at the same logical point in the stream), no events are lost and no events are double-counted. This is exactly-once processing semantics.