Start practising

SYSTEM DESIGN · CONCEPTS · RUNNING IT IN PRODUCTION · EPISODE 41

Batch vs Stream Processing Basics

Two ways to process large amounts of data: batch jobs over stored data (MapReduce, Spark, data warehouses) and stream processing over events as they arrive (Kafka, Flink), windows and event time vs processing time, exactly-once and late events, lambda vs kappa architectures, and how to choose in a design.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
1 / 10

Press play to hear the narration, or step through at your own pace. Every picture here is the same one used in the YouTube video.

Notes

Events

Everything starts as events: a user clicked, an order was placed, a payment went through, each with a timestamp. A log like Kafka stores them durably, in order, and lets many consumers read the same events independently. What you do next is either batch or stream.

An event log
  • events: something happened,
  • with a timestamp
  • {user: 42, action: "order",
  • amount: 999, ts: 10:42:07}
  • Kafka keeps them in order,
  • durably, for days
  • many consumers read the
  • same events independently

Batch

Batch processing stores the events, in a data lake or warehouse, and processes big chunks on a schedule: every night, crunch yesterday. Tools are Spark, MapReduce or plain SQL in a warehouse. It's simple, efficient and easy to re-run if you find a bug. The cost is latency: results arrive hours later.

Batch processing
  • collect data, then process a
  • whole chunk: last day, last hour
  • MapReduce, Spark, SQL in a
  • warehouse (BigQuery, Snowflake)
  • + simple, efficient, easy to
  • re-run when code has a bug
  • − results arrive hours later
  • e.g. daily revenue, invoices,
  • model training, search indexes

Stream

Stream processing runs a job continuously that handles each event within seconds of it happening. Tools are Flink, Kafka Streams or Spark's streaming mode. You get fraud alerts, live dashboards, surge pricing and trending topics in seconds. The price is complexity: state, failures and ordering are much harder than in batch.

Stream processing
  • a job runs forever, handling
  • events as they arrive
  • Flink, Kafka Streams,
  • Spark Structured Streaming
  • + results in seconds
  • − state, failures and ordering
  • are harder
  • e.g. fraud checks, live
  • dashboards, surge pricing,
  • trending topics

Windows

A stream never ends, so 'total orders' has no final answer. Instead you aggregate over windows: fixed five-minute buckets, a sliding last-ten-minutes window, or sessions that end after a quiet period. 'More than five payments on one card in ten minutes' is a sliding-window fraud rule.

Windows
  • streams never end, so group
  • events into windows:
  • tumbling: 10:00–10:05, 10:05–…
  • sliding: last 5 min, every 1 min
  • session: until 30 min of quiet
  • "orders per city per minute"
  • "payments per card in the
  • last 10 minutes > 5 → flag"

Late events

A subtle problem: a phone goes offline and sends its two-minutes-past events at nine past. Event time, when it happened, differs from processing time, when it arrived. Count by event time, or reports drift. Stream engines use watermarks, an estimate of 'all events up to now have probably arrived', and a policy for events that come later still.

Event time vs arrival time
  • a phone goes offline, sends
  • its 10:02 events at 10:09
  • event time: when it happened
  • processing time: when it arrived
  • count by event time, or
  • reports are wrong
  • watermarks: "probably all
  • events up to 10:05 are in"
  • late events: update or drop

Exactly once

When a stream job crashes, it restarts from a checkpoint and replays some events. Modern engines save their state and input positions together, so results inside the pipeline are effectively exactly once. But side effects outside it, like sending an email or charging a card, still need to be idempotent.

Exactly once?
  • failures cause replays →
  • duplicates
  • engines checkpoint state and
  • input offsets together
  • (Flink, Kafka transactions)
  • effectively exactly-once
  • INSIDE the pipeline
  • external side effects still
  • need idempotency (ep. 29)

Architectures

Two classic architectures. Lambda runs both: an accurate batch layer and a fast stream layer, merged at query time, but you maintain the same logic twice. Kappa streams everything and fixes mistakes by replaying the log through new code. In practice, many teams stream for live features and use warehouse SQL for reports.

Lambda vs kappa
  • lambda: batch layer (accurate,
  • slow) + speed layer (fast,
  • approximate), merged
  • − two codebases to keep equal
  • kappa: stream only; to fix a
  • bug, replay the log through
  • new code
  • + one codebase
  • many teams: stream for live,
  • warehouse SQL for reports

Say it like this

In an interview: send events into Kafka. Anything that needs results in seconds, like fraud checks, live counters or alerts, goes to a stream job with windows on event time. Anything that can wait, like reports, billing and training data, lands in a data lake and runs as batch. Keep sinks idempotent and the log replayable.

In the interview
  • events → Kafka
  • needs seconds? (fraud, live
  • counts, alerts) → Flink job,
  • windows, event time
  • can wait? (reports, billing,
  • ML features) → lake + Spark /
  • warehouse, nightly
  • idempotent sinks, replayable log

Interview questions

What is the difference between batch and stream processing?

Batch processing runs over a bounded set of stored data on a schedule, producing results with hours of latency. Stream processing handles unbounded data continuously, event by event, producing results within seconds.

When would you choose stream processing?

When results lose value quickly: fraud detection, live dashboards and counters, alerting, surge pricing, trending topics and real-time personalisation.

When is batch processing the better choice?

For reports, billing, reconciliation, machine-learning training and backfills, where results can wait and simplicity, efficiency and easy re-runs matter more than latency.

What are windows in stream processing?

Ways to group an unbounded stream into finite chunks for aggregation: tumbling (fixed, non-overlapping), sliding (overlapping) and session windows (bounded by inactivity gaps).

What is the difference between event time and processing time?

Event time is when the event actually happened; processing time is when the system received it. Delays and offline devices make them differ, so correct aggregations use event time with watermarks for late data.

How do stream processors achieve exactly-once results?

By checkpointing operator state together with input offsets and using transactional or idempotent outputs, so after a failure they replay from a consistent point without double counting.

What are lambda and kappa architectures?

Lambda combines a batch layer for accurate results with a speed layer for low-latency approximations. Kappa uses only a streaming pipeline over a replayable log and reprocesses history by replaying it.

Why put a log like Kafka in the middle?

It decouples producers from consumers, stores events durably and in order, lets many independent consumers (stream jobs, lake loaders) read the same data, and allows replay after bugs.