Flink Concept 2: The Problem of Time and Watermarks

Event Time vs. Processing Time

In streaming, there are two clocks that are almost never in sync:

1. Event Time: The timestamp embedded in the event itself — when the event actually happened in the real world (e.g., the timestamp on the sensor reading or the transaction record). This is almost always what you care about for business logic.
2. Processing Time: The wall-clock time on the Flink server at the moment it processes the event. This is always "now" from the server's perspective.

Due to network delays, mobile devices going offline and syncing later, and distributed system realities, data almost always arrives out of order. An event that happened at 10:00:00 might arrive at the Flink processor at 10:00:05, after an event that happened at 10:00:03.

If you use Processing Time for your windows, your results are fast but potentially wrong — you're grouping events by when they arrived, not when they happened. If you use Event Time, your results are correct, but you face a new problem: how do you know when you've received all the data for a given time window?

Watermarks: A Heuristic for Completeness

If you are calculating a tumbling window for 10:00–10:05 based on Event Time, how do you know when to close that window and emit the result? You can't wait forever for late data.

A Watermark is Flink's mechanism for answering this question. A Watermark with timestamp T is a signal that flows through the stream alongside the data, asserting: "I believe I have now seen all events that occurred up to time T. Any window that ends before T can be safely closed and its result emitted."

Watermarks are generated by a WatermarkStrategy that you configure. A common strategy is:

Watermark = max(event_time_seen_so_far) - allowed_lateness

For example, if allowed_lateness = 5 seconds, and the latest event time seen is 10:00:30, the current watermark is 10:00:25. This means Flink will wait up to 5 seconds for late-arriving data before closing a window.

The trade-off is fundamental: A larger allowed_lateness gives you more correct results (fewer events missed) but increases the latency before you see output. A smaller value gives you faster output but may miss late-arriving events. There is no universally correct answer — it depends on your data's latency characteristics and your business tolerance for late data.