The idea: event time, processing time and a watermark
A stream job reads records that never stop, for example clicks or sensor readings from a Kafka topic. To count or add them up, it cuts the stream into windows of time, such as "every 10 seconds". But which time? Each record carries its event time t: when it happened. The job sees it later, at its processing time p. Networks, retries and phones that were offline make records arrive out of order. On this page e4 happened at t = 3 but arrives after e3 (t = 7), and e10 (t = 9) arrives last of all.
The job can never be sure that all records of a window have arrived. A watermark is its guess, sent along with the stream: "W = 11" means "I expect no more records with t < 11". Here the watermark is the largest event time seen minus 3 s (bounded out-of-orderness of 3 s). When the watermark passes the end of a window, the window fires: it sends its result to the output.
Three kinds of windows
A window assigner decides which windows a record belongs to. The page uses the same 10 records with each kind.
| window | Flink assigner | windows of e3 (t = 7) |
|---|---|---|
| tumbling 10 s | TumblingEventTimeWindows.of(10 s) | [0,10): one window per record |
| sliding 10 s every 5 s | SlidingEventTimeWindows.of(10 s, 5 s) | [0,10) and [5,15): size / slide = 2 windows |
| session, gap 4 s | EventTimeSessionWindows.withGap(4 s) | starts as [7,11), then merges into [1,11) |
A session window has no fixed start. Each record opens [t, t + gap). Windows that overlap or touch merge. A session closes after a gap of 4 s with no record. In Demo: session windows, gap 4 s, e1, e2, e3 and e4 merge into [1,11), and e5, e6 and e8 into [12,21).
The watermark: when is a window complete?
In Demo: tumbling windows, the watermark fires them, the watermark moves −2, 1, 4, 4, 9, 11 … as records arrive. It never goes back: when e4 (t = 3) comes, the watermark stays at 4. e6 (t = 14) lifts it to 14 − 3 = 11 ≥ 10, so [0,10) fires with n = 4, sum = 10. e9 (t = 24) lifts it to 21, and [10,20) fires with n = 3, sum = 8. [20,30) is still open at the end: nothing has pushed the watermark past 30.
A record that is out of order is not the same as one that is late. e4 arrives after e3, but the watermark is only 4 when it comes, so [0,10) is still open and counts it (Demo: out of order but not late). The delay sets the trade-off: a bigger delay waits longer for stragglers, so results come out later; a smaller one gives faster results and more late records.
Late events and allowed lateness
A record is late when its window already fired and was cleared. e7 (t = 6) arrives when W = 11, after [0,10) fired. With allowed lateness 0, the window state is cleared as soon as it fires, so e7 is dropped (Demo: a late event is dropped). Flink can send dropped records to a side output (sideOutputLateData) instead of losing them.
With allowedLateness(5 s) the state of [0,10) is kept until W ≥ 10 + 5. e7 now gets in, and the window fires again at once with n = 5, sum = 16. The output then has two rows for [0,10): the reader must treat the second as an update of the first. At W = 21 the state is cleared, so e10 (t = 9) is still dropped (Demo: allowed lateness 5 s, the window fires again). With sliding windows a record can be late for one window and on time for another: e7 misses [0,10) but still counts in [5,15) (Demo: sliding windows, 10 s every 5 s).
Processing-time windows
A processing-time window uses the clock of the machine instead of t. It needs no watermark, fires on time and drops nothing. But it answers a different question: "what arrived between 10 s and 20 s", not "what happened then". In Demo: processing time, delayed events in the wrong window, [10,20) gets e7 even though it happened at t = 6, and e10 (t = 9) lands in [20,30). Run the same records again at another speed and the results change. With event time, the records with t < 10 are six records with sum 17, whatever the arrival order. The event-time demos count four or five of them in [0,10), because e7 and e10 come too late.
The same ideas in Spark Structured Streaming
| idea | Flink DataStream | Spark Structured Streaming |
|---|---|---|
| watermark, 3 s delay | WatermarkStrategy.forBoundedOutOfOrderness(3 s) | withWatermark("t", "3 seconds") |
| tumbling | TumblingEventTimeWindows.of(10 s) | window(t, "10 seconds") |
| sliding | SlidingEventTimeWindows.of(10 s, 5 s) | window(t, "10 seconds", "5 seconds") |
| session | EventTimeSessionWindows.withGap(4 s) | session_window(t, "4 seconds") |
| late records | allowedLateness, side output | no separate lateness: the watermark delay is the limit; older rows are dropped |
| when results come out | trigger fires per window | output mode append: once, when the watermark passes the end; update: every micro-batch |
Spark works in micro-batches: it computes the watermark at the end of each batch and uses it in the next one. Flink handles one record at a time, as on this page. Why the job can restart after a crash without counting twice is a separate story: see Delivery Semantics and the Chandy–Lamport snapshot behind Flink's checkpoints. For batch jobs on fixed data, see Spark RDD stages.
See also Lambda vs Kappa architecture, which compares a batch layer plus a speed layer with one stream job that fixes bugs by replaying a log.
What the page leaves out
Keys: real jobs window per key (keyBy), each key with its own windows. Parallel sources: the watermark of an operator is the smallest watermark of its inputs, and an idle input can hold it back (withIdleness). Periodic watermarks: Flink emits them every 200 ms (autoWatermarkInterval), not after every record. Custom triggers and evictors, early firings, global and count windows, and Flink's 1 ms shifts (it sets W = max − 3 s − 1 ms and fires at end − 1 ms, which cancel out). Spark's per-batch watermark, state store and the complete output mode. Checkpoints and exactly-once output.