The idea: event time, processing time and a watermark

A stream job reads records that never stop, for example clicks or sensor readings from a Kafka topic. To count or add them up, it cuts the stream into windows of time, such as "every 10 seconds". But which time? Each record carries its event time t: when it happened. The job sees it later, at its processing time p. Networks, retries and phones that were offline make records arrive out of order. On this page e4 happened at t = 3 but arrives after e3 (t = 7), and e10 (t = 9) arrives last of all.

A chart with event time across and processing time up. The ten records e1 to e10 are dots. A dashed diagonal marks records that arrive the moment they happen; every dot lies above it, some far above: e4 at t 3, p 9; e7 at t 6, p 16; e10 at t 9, p 26. A blue staircase shows the watermark, 3 seconds behind the largest event time seen so far. e7 and e10 lie left of the watermark when they arrive, so they are late
Records arrive after they happen, some much later. The watermark (blue) trails the largest event time by 3 s; e7 and e10 arrive behind it.

The job can never be sure that all records of a window have arrived. A watermark is its guess, sent along with the stream: "W = 11" means "I expect no more records with t < 11". Here the watermark is the largest event time seen minus 3 s (bounded out-of-orderness of 3 s). When the watermark passes the end of a window, the window fires: it sends its result to the output.

Three kinds of windows

A window assigner decides which windows a record belongs to. The page uses the same 10 records with each kind.

One event time axis from 0 to 30 seconds with the ten records as dots. Below it three rows. Tumbling: [0,10), [10,20), [20,30) side by side. Sliding 10 s every 5 s: [-5,5), [0,10), [5,15), [10,20), [15,25), [20,30), overlapping. Session with gap 4 s: [1,11), [12,21), [24,28), separated by gaps with no records
Tumbling windows split time evenly, sliding windows overlap so each record counts twice, and session windows follow the gaps in the data.
windowFlink assignerwindows of e3 (t = 7)
tumbling 10 sTumblingEventTimeWindows.of(10 s)[0,10): one window per record
sliding 10 s every 5 sSlidingEventTimeWindows.of(10 s, 5 s)[0,10) and [5,15): size / slide = 2 windows
session, gap 4 sEventTimeSessionWindows.withGap(4 s)starts as [7,11), then merges into [1,11)

A session window has no fixed start. Each record opens [t, t + gap). Windows that overlap or touch merge. A session closes after a gap of 4 s with no record. In Demo: session windows, gap 4 s, e1, e2, e3 and e4 merge into [1,11), and e5, e6 and e8 into [12,21).

The watermark: when is a window complete?

In Demo: tumbling windows, the watermark fires them, the watermark moves −2, 1, 4, 4, 9, 11 … as records arrive. It never goes back: when e4 (t = 3) comes, the watermark stays at 4. e6 (t = 14) lifts it to 14 − 3 = 11 ≥ 10, so [0,10) fires with n = 4, sum = 10. e9 (t = 24) lifts it to 21, and [10,20) fires with n = 3, sum = 8. [20,30) is still open at the end: nothing has pushed the watermark past 30.

A record that is out of order is not the same as one that is late. e4 arrives after e3, but the watermark is only 4 when it comes, so [0,10) is still open and counts it (Demo: out of order but not late). The delay sets the trade-off: a bigger delay waits longer for stragglers, so results come out later; a smaller one gives faster results and more late records.

Late events and allowed lateness

A record is late when its window already fired and was cleared. e7 (t = 6) arrives when W = 11, after [0,10) fired. With allowed lateness 0, the window state is cleared as soon as it fires, so e7 is dropped (Demo: a late event is dropped). Flink can send dropped records to a side output (sideOutputLateData) instead of losing them.

A watermark axis for window [0,10) with allowed lateness 5 s. Before W 10: the window is open and collects records. From W 10: it fires once, with n 4 and sum 10. From W 10 up to 15: a late record still gets in and the window fires again; e7 arrives at W 11 and gives n 5, sum 16. From W 15 on: the state is cleared and a record such as e10, arriving at W 21, is dropped
With allowed lateness 5 s, window [0,10) fires once W reaches 10, fires again for each late record until W = 15, and then forgets.

With allowedLateness(5 s) the state of [0,10) is kept until W ≥ 10 + 5. e7 now gets in, and the window fires again at once with n = 5, sum = 16. The output then has two rows for [0,10): the reader must treat the second as an update of the first. At W = 21 the state is cleared, so e10 (t = 9) is still dropped (Demo: allowed lateness 5 s, the window fires again). With sliding windows a record can be late for one window and on time for another: e7 misses [0,10) but still counts in [5,15) (Demo: sliding windows, 10 s every 5 s).

Processing-time windows

A processing-time window uses the clock of the machine instead of t. It needs no watermark, fires on time and drops nothing. But it answers a different question: "what arrived between 10 s and 20 s", not "what happened then". In Demo: processing time, delayed events in the wrong window, [10,20) gets e7 even though it happened at t = 6, and e10 (t = 9) lands in [20,30). Run the same records again at another speed and the results change. With event time, the records with t < 10 are six records with sum 17, whatever the arrival order. The event-time demos count four or five of them in [0,10), because e7 and e10 come too late.

The same ideas in Spark Structured Streaming

ideaFlink DataStreamSpark Structured Streaming
watermark, 3 s delayWatermarkStrategy.forBoundedOutOfOrderness(3 s)withWatermark("t", "3 seconds")
tumblingTumblingEventTimeWindows.of(10 s)window(t, "10 seconds")
slidingSlidingEventTimeWindows.of(10 s, 5 s)window(t, "10 seconds", "5 seconds")
sessionEventTimeSessionWindows.withGap(4 s)session_window(t, "4 seconds")
late recordsallowedLateness, side outputno separate lateness: the watermark delay is the limit; older rows are dropped
when results come outtrigger fires per windowoutput mode append: once, when the watermark passes the end; update: every micro-batch

Spark works in micro-batches: it computes the watermark at the end of each batch and uses it in the next one. Flink handles one record at a time, as on this page. Why the job can restart after a crash without counting twice is a separate story: see Delivery Semantics and the Chandy–Lamport snapshot behind Flink's checkpoints. For batch jobs on fixed data, see Spark RDD stages.

See also Lambda vs Kappa architecture, which compares a batch layer plus a speed layer with one stream job that fixes bugs by replaying a log.

What the page leaves out

Keys: real jobs window per key (keyBy), each key with its own windows. Parallel sources: the watermark of an operator is the smallest watermark of its inputs, and an idle input can hold it back (withIdleness). Periodic watermarks: Flink emits them every 200 ms (autoWatermarkInterval), not after every record. Custom triggers and evictors, early firings, global and count windows, and Flink's 1 ms shifts (it sets W = max − 3 s − 1 ms and fires at end − 1 ms, which cancel out). Spark's per-batch watermark, state store and the complete output mode. Checkpoints and exactly-once output.

On the mind map

Open the Distributed Systems mind mapDistributedSystemsReplication: copies of data · what reads seeReplicationPartitioning: split data over nodesPartitioningTime and membership: order events, know who is aliveTime and membershipConsensus: agree on one value or logConsensusTransactions: one change across many servicesTransactionsMessaging: send events between systemsMessagingKafka internals: a replicated append-only logKafka internalsRedis internals: one thread, data in memoryRedis internalsKubernetes: reconcile desired stateKubernetesResilience and speed: stay up and fast under loadResilience and speedKafka message pathKafka message pathbatch, partition, replicate, commit offsetStream Windows & WatermarksWindow fires when watermark passes its endLambda vs KappaLambda vs Kappatwo code paths vs replay one logKafka rebalanceKafka rebalancepartitions move; duplicates appearKafka log segmentsKafka log segmentsfiles by base offset, sparse indexKafka replicationKafka replicationISR and high watermark; epochs truncateKafka log compactionKafka log compactionkeep the last value per keyKafka KRaftKafka KRaftRaft quorum stores cluster metadataKafka zero copyKafka zero copysendfile: page cache to socketOpen the Distributed Systems mind mapDistributed Systemsa replicated append-only logKafka internalsKafka message pathKafka message pathbatch, partition, replicate, commit offsetStream Windows & WatermarksWindow fires when watermark passes its endLambda vs KappaLambda vs Kappatwo code paths vs replay one logKafka rebalanceKafka rebalancepartitions move; duplicates appearKafka log segmentsKafka log segmentsfiles by base offset, sparse indexKafka replicationKafka replicationISR and high watermark; epochs truncateKafka log compactionKafka log compactionkeep the last value per keyKafka KRaftKafka KRaftRaft quorum stores cluster metadataKafka zero copyKafka zero copysendfile: page cache to socket
This page on the Distributed Systems mind map · 88 pages in 10 branches · open it to see the whole map and test yourself in recall mode.