Start practising

SYSTEM DESIGN · TECHNOLOGIES · 7 OF 12

Kafka: The Distributed Log for Event Streams and Decoupled Services

How Apache Kafka works as a replicated, partitioned, append-only log: topics, partitions and offsets, keys and ordering, why sequential disk and batching make it fast, leaders, in-sync replicas and acks=all, consumer groups and rebalancing, retention and compaction, exactly-once, KRaft, where it hurts and how to place it in an interview design.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
1 / 16

Press play to hear the narration, or step through at your own pace.

Notes

What it is

Kafka is a distributed commit log. Producers append events to the end; consumers read from a position called an offset. Unlike a classic queue, reading does not delete anything. Events stay for a retention period, days or even forever, so a new service can start from the beginning and a broken one can rewind. Confluent and Amazon MSK run it for you, and Redpanda speaks the same protocol.

Kafka in one line
  • a distributed commit log:
  • producers append events,
  • consumers read by offset
  • events are kept (days, or
  • forever), not deleted on read
  • managed: Confluent, MSK,
  • Aiven; compatible: Redpanda,
  • WarpStream

Data model

The model: a topic, like orders, is split into partitions. Each partition is an ordered, append-only sequence of records, numbered by offset. A record has a key, a value, a timestamp and headers. The producer hashes the key to pick a partition, so all events for order nine eight one land in the same partition, in order. Remember this sentence: Kafka orders events within a partition, never across the whole topic.

Topics, partitions, offsets
topic: orders (3 partitions)

p0: [0][1][2][3][4][5] ...
p1: [0][1][2][3] ...
p2: [0][1][2][3][4] ...

record = key, value, timestamp,
         headers
partition = hash(key) % 3
order is kept within a partition only

Producing

Producers group records into batches per partition and send each batch to the broker that leads that partition. Waiting a few milliseconds to fill a batch, and compressing it, multiplies throughput. Records without a key are spread for balance, so they lose ordering. Idempotent producers are the default now: each batch carries a sequence number, so a retry after a timeout never writes a duplicate.

Producer settings
  • key → partition (same key,
  • same partition, same order)
  • no key → spread for balance
  • batch.size + linger.ms:
  • group records, fewer requests
  • compression: lz4 / zstd
  • enable.idempotence=true:
  • retries never duplicate
  • (default since 3.0)

Why it is fast

Why is it so fast? Every write is a sequential append to a file, which even spinning disks do quickly. Recent data sits in the operating system's page cache, so consumers that keep up read from memory. Brokers send file data straight to the network socket without copying it through the application. Batching and compression shrink the work further, and partitions spread it across many brokers.

Why it is fast
  • • append-only: sequential disk
  • writes, no random seeks
  • • OS page cache holds recent
  • data; most reads hit RAM
  • • zero-copy: file → socket
  • without passing through
  • the application
  • • batches + compression
  • • partitions spread load
  • across brokers

Replication

Each partition is replicated, usually three times. One broker is the leader and takes the writes; followers fetch from it. Replicas that are caught up form the in-sync set. With acks all and a minimum of two in-sync replicas, a write is confirmed only when at least two brokers have it, so losing any one broker loses nothing. If the leader dies, an in-sync follower takes over.

Durability settings
  • replication.factor = 3
  • leader takes writes, followers
  • fetch from it
  • ISR: in-sync replicas that
  • are caught up
  • acks=all + min.insync.
  • replicas=2 → survives the
  • loss of any one broker
  • leader dies → an ISR member
  • takes over

Consumer groups

Consumers join a group. Within a group, each partition is assigned to exactly one consumer, so the work is shared and per-key order is kept. Here two billing consumers split three partitions. A separate analytics group reads the same events independently, with its own offsets. One important consequence: partitions cap parallelism. Ten consumers on three partitions means seven sit idle, so choose the partition count with growth in mind.

Consumer groups
  • group = one logical reader
  • each partition → exactly one
  • consumer in the group
  • more consumers than
  • partitions → some sit idle
  • each group tracks its own
  • committed offsets
  • billing and analytics read the
  • same events independently

Offsets + delivery

A consumer records progress by committing its offset. Commit after processing and a crash means some events are processed again: that is at-least-once, the usual choice, so make handlers idempotent, as in Concepts episode 29. Commit first and a crash can skip events: at-most-once. Kafka transactions give exactly-once for read, process and write pipelines that stay inside Kafka, such as stream processing jobs.

Delivery guarantees
  • consumer commits its offset
  • to an internal topic
  • commit after processing
  • → at-least-once (dupes on
  • crash): make it idempotent
  • commit before processing
  • → at-most-once (may skip)
  • exactly-once: transactions,
  • for Kafka-to-Kafka pipelines

Retention

Retention is by time or size, seven days by default; old log segments are simply deleted. Or use log compaction, which keeps only the latest record for each key, so a topic becomes a durable table of current values, like each user's settings, that a new service can load from the start. Tiered storage moves older segments to object storage, so you can keep months cheaply.

Keeping data
  • retention.ms (default 7 days)
  • or retention.bytes per
  • partition: old segments
  • are deleted
  • cleanup.policy=compact:
  • keep the latest record per
  • key forever (a changelog,
  • e.g. user settings)
  • tiered storage: old data in S3

Rebalancing + KRaft

Two operational points. When a consumer joins, leaves or stops responding, the group rebalances and partitions move; newer cooperative protocols move only what they must, instead of pausing everyone. And cluster metadata is now kept by a small Raft quorum of controllers called KRaft; ZooKeeper support was removed in Kafka four. Also note that adding partitions later changes which partition a key maps to, which breaks per-key order for a moment.

Operations to know
  • rebalance: consumer joins or
  • leaves → partitions are
  • reassigned; cooperative
  • protocol avoids full pauses
  • KRaft: Raft-based controller
  • quorum stores metadata;
  • ZooKeeper removed in 4.0
  • adding partitions changes
  • key → partition mapping

Great at

Kafka is great at decoupling: the order service publishes one event and billing, email, search and analytics each consume it without knowing about each other. It handles huge volumes of clickstream, logs and metrics. It is the standard pipe for change data capture from databases. It feeds stream processors like Flink. And because history is kept, you can replay events to rebuild a cache or fix a bug's damage.

Where Kafka shines
  • • decoupling services: one
  • event, many subscribers
  • • activity, clickstream, logs,
  • metrics at huge volume
  • • change data capture
  • (Debezium → Kafka)
  • • stream processing (Flink,
  • Kafka Streams)
  • • replay to rebuild state

Where it hurts

Where it hurts. It is not a classic task queue: no per-message acknowledgement, no delayed delivery, no priorities, and retries mean building retry topics yourself. Newer share groups add queue-like consumption, but SQS or RabbitMQ are simpler for jobs. Order exists only within a partition, and one slow event blocks everything behind it in that partition. A hot key overloads one partition. Running clusters yourself is real work.

Pain points
  • • no per-message ack, delay,
  • priority or easy retries
  • (classic queues do this)
  • • order only per partition
  • • parallelism capped by
  • partition count
  • • one slow event blocks its
  • partition (head of line)
  • • hot key → hot partition
  • • heavy to run yourself

Use it when

Use Kafka when several services must react to the same stream, when volume is high and order per key matters, or when you need replay and change data capture. Avoid it for simple background jobs that need retries, delays and per-message acknowledgement, where SQS or RabbitMQ fit better, for low volume where it is overkill, and for request and response calls. Concepts episode 28 compares queues and pub-sub.

Use it when / avoid it when
  • USE when
  • many services react to one
  • event stream
  • high volume, ordered by key
  • you need replay or CDC
  • AVOID when
  • simple background jobs with
  • retries/delays → SQS, Rabbit
  • low volume, small team
  • request/response calls

Numbers

Numbers to have ready. A well-tuned broker sustains hundreds of megabytes a second, and end-to-end latency is typically a few milliseconds to a few tens of milliseconds, depending on batching. Use a replication factor of three with a minimum of two in-sync replicas. Default retention is seven days and the default maximum message size is about one megabyte, so put big payloads in object storage and send a pointer.

Numbers worth knowing
throughput per broker    100s of MB/s
end-to-end latency       ~5-50 ms
replication.factor       3
min.insync.replicas      2
default retention        7 days
default max message      ~1 MB
partitions per cluster   up to ~100k+
                         (with KRaft)
consumers per partition  1 per group

Say it like this

How to say it. The order service writes the order and publishes an order placed event through the outbox pattern, from Concepts episode 33, so the database and Kafka never disagree. The key is order id, so each order's events stay in sequence. Twenty-four partitions for headroom, replication factor three, acks all. Billing and email are separate consumer groups, and their handlers are idempotent because delivery is at least once.

In the interview
  • "Order service writes the
  • order, then publishes
  • OrderPlaced to Kafka via the
  • outbox. Key = order_id so
  • each order’s events stay in
  • order. 24 partitions, RF 3,
  • acks=all. Billing and email
  • are separate groups;
  • handlers are idempotent."

Interview questions

How does Kafka guarantee ordering?

Only within a partition. Records with the same key are hashed to the same partition and appended in order, so per-key order holds, but there is no total order across partitions of a topic.

What is a consumer group?

A set of consumers that share a topic’s partitions, with each partition assigned to exactly one consumer in the group. Different groups read the same data independently, each with its own committed offsets.

How do you make Kafka writes durable?

Use a replication factor of three, set acks=all on the producer and min.insync.replicas=2 on the topic, so a write is only acknowledged once at least two in-sync replicas have it and any single broker can fail without data loss.

Why is Kafka so fast?

It appends sequentially to log files, relies on the operating system page cache, batches and compresses records, sends data with zero-copy from file to socket, and spreads partitions across brokers.

What delivery guarantee does Kafka give?

By default at-least-once when consumers commit offsets after processing, so handlers must be idempotent. Exactly-once is available for Kafka-to-Kafka processing through idempotent producers and transactions.

How many partitions should a topic have?

Enough for the peak consumer parallelism and throughput you expect, since each partition is consumed by at most one consumer per group; adding partitions later changes the key-to-partition mapping, so plan headroom up front.

What is log compaction?

A cleanup policy that keeps at least the latest record for each key and removes older ones, turning a topic into a durable changelog of current state that new consumers can load from the beginning.

Kafka or a message queue like SQS or RabbitMQ?

Kafka for high-volume event streams with many independent consumers, ordering per key and replay. A traditional queue for background jobs that need per-message acknowledgement, retries, delays or priorities.

Does Kafka still need ZooKeeper?

No. Kafka now stores cluster metadata in its own Raft-based controller quorum called KRaft, and ZooKeeper mode was removed in Kafka 4.0.