The idea: two views, or one log you can read again
A site wants to know how many times each page was viewed. The views come as a stream of events. The answer must be fresh (include the last second) and right (include every old event, counted with the right rule). Two well-known designs give both. Lambda runs two jobs: a slow batch job over all the data and a fast stream job over the newest data, and adds their results. Kappa runs one stream job over a log that keeps every event, and reads the log again when the rule changes.
On this page 12 events e1 … e12 are views of page A or page B. Three of them (e3, e8, e12) come from a bot. The job should not count bot views, so the right answer after all 12 is A = 5, B = 4. Code v1 has a bug: it counts bot views and would say A = 7, B = 5. Code v2 is the fix.
Lambda: batch layer, speed layer, serving layer
Nathan Marz named the Lambda architecture in 2011. It has three layers:
- Batch layer. Every event is appended to the master dataset, for example files in HDFS. Nothing is ever changed or deleted. Now and then a batch job (MapReduce or Spark) reads all of it and computes the batch view from scratch. Here it runs after every 5th event; in real life it may run every night and take hours.
- Speed layer. A stream job counts each event as it comes into a realtime view. It only covers the events since the last batch run. After a batch run it throws those counts away, because the batch view has them now.
- Serving layer. A query adds the two: answer = batch view + realtime view.
In Demo: Lambda, batch view + realtime view at query time (code v2), the batch job runs twice, after e5 and e10. At the end the batch view covers e1–e10 with A = 4, B = 4, the realtime view covers e11–e12 with A = 1, B = 0, and the query gives A = 5, B = 4.
A bug fix in Lambda
The bug must be fixed twice: once in the batch job and once in the speed job. They are often written with different tools, so it is two pieces of code that must give the same numbers. In Demo: Lambda, a bug fix in both code paths, the fix ships after e8. At that moment the batch view (e1–e5, v1) says A = 3, B = 2 and the realtime view (e6–e8, v1) says A = 2, B = 1, so the query says A = 5 while the right A is 3. The fix changes nothing that is already counted. The query stays wrong until the batch job runs again after e10: it recomputes e1–e10 with v2 from the raw data and gives A = 4, B = 4. From then on the answer is right. Marz calls this human fault tolerance: since the raw data is never changed, rerunning the batch job repairs any bug.
If only the batch job is fixed, the two paths drift apart (Demo: Lambda, fix only the batch code: the paths drift). After the batch run at e10 the answer is right, but the speed job still runs v1 and counts the bot view e12. The query ends with B = 5 instead of 4, and it would jump back to 4 at the next batch run. Answers that change on their own like this are hard to debug.
Kappa: one log, replay to reprocess
Jay Kreps proposed the Kappa architecture in 2014 as a reply. Keep the raw events in a log such as a Kafka topic, with a long retention. One stream job reads the log in order, keeps a table, and commits its offset. There is no batch layer. In Demo: Kappa, one log and one stream job, job v2 reads offsets 0–11 and its table says A = 5, B = 4.
To fix a bug, replay the log (Demo: Kappa, a bug fix by replaying the log):
- Start job v2 in a new consumer group (
counts-v2) at offset 0. It writes a new table,counts_v2. Job v1 keeps serving. - Job v2 reads offsets 0–7 again (8 events) as fast as it can and builds A = 3, B = 3. Job v1 had A = 5, B = 3.
- When v2 has caught up (lag 0), switch the serving layer to
counts_v2. - Stop v1 and drop
counts_v1. New events e9–e12 go to v2 only, and the end result is A = 5, B = 4.
The replay only works if the log still has the old events. In Demo: Kappa, retention too short for the replay the topic keeps only the last 6 events (think of a short retention.ms). When the fix ships, offsets 0 and 1 are gone, so v2 starts at offset 2. It misses e1 (A) and e2 (B) and ends with A = 4, B = 3: wrong for good. Kappa needs retention as long as the history you may want to recompute, or a copy of the log in cheap storage. For a log that keeps only the newest value per key, see Kafka log compaction.
The cost of two code paths
| Lambda | Kappa | |
|---|---|---|
| raw data kept in | master dataset (HDFS, S3) | the log (Kafka topic, long retention) |
| code to write and test | 2: batch job + speed job, must agree | 1 stream job |
| systems to run | batch engine + stream engine + a store that merges views | log + stream engine + a table |
| fixing a bug | change both jobs; history heals at the next batch run | run v2 from offset 0, switch, stop v1 |
| what it needs | time and machines for full recomputes | a log that keeps everything; a job fast enough to replay it |
| good when | the batch side needs work a stream job does badly (big joins, ML training) | the logic fits a stream job; one team owns one code base |
Today many teams run one engine for both (Spark Structured Streaming, Apache Flink, Apache Beam), so the "two code paths" can share code even in a Lambda setup. How a stream job groups events by time and handles late ones is on Stream Windows and Watermarks. The batch side is MapReduce or Spark stages. Keeping every event and rebuilding state from it is also the idea behind event sourcing.
What the page leaves out
Keys, partitions and many parallel tasks; time windows and late events. A real batch run takes hours, so new events arrive during it and the speed layer must keep the events since the batch run started. The realtime view is often approximate (sampling, sketches). Exactly-once output and idempotent writes during a replay (see Delivery Semantics). Tiered storage that moves old log segments to object storage, and replaying from a data lake instead of the log. Serving stores such as Druid, Cassandra or Pinot and how they swap tables.