Notes
The problem
The problem: checkout calls the email service directly and waits three seconds for the provider. If email is down, checkout fails. With a queue in between, checkout writes a message in a few milliseconds and moves on. If the workers are down, messages wait. A spike is buffered instead of dropped. You pay with eventual processing, which is fine for an email, not for a payment authorisation.
- order API waits 3 s for the
- email provider → slow checkout
- email down → checkout fails
- with a queue:
- producer returns in ms
- consumer down → work waits
- spikes are buffered
- scale consumers separately
- cost: eventual, not instant
Inside the broker
Inside the broker, a topic is split into partitions, and each partition is an append-only log file on disk, with every message getting the next offset number. The producer chooses a partition by hashing a key, like the order id, so all messages for one order land in the same partition, in order. Appending sequentially to files and reading through the operating system's page cache is why one broker can move hundreds of megabytes a second.
- topic "order-placed" split
- into partitions (here 3)
- partition = append-only log
- on disk, offsets 0, 1, 2 ...
- producer picks partition by
- hash(key) → same key, same
- partition, same order
- sequential disk writes +
- page cache → very fast
Queue vs log
There are two designs. A classic queue, like SQS or RabbitMQ, deletes each message once it's acknowledged, tracks acks per message, and lets workers compete for jobs, which is ideal for tasks. A log, like Kafka, keeps messages for days whether read or not; each consumer just remembers its offset. That allows replay and many independent readers, with order per partition. The catch: parallelism is capped at the number of partitions.
- queue (SQS, RabbitMQ):
- message deleted when acked
- per-message ack + redelivery
- workers compete for jobs
- log (Kafka, Kinesis, Pulsar):
- messages kept for days
- consumer tracks an offset
- replay, many readers
- order within a partition
- parallelism = partitions
Consumer groups
Consumers join a group. Within a group, each partition is assigned to exactly one consumer, which keeps per-partition order. Three partitions and two workers means one worker takes two partitions. Add a fourth worker to three partitions and it sits idle. A different group, say analytics, reads the same messages completely independently, with its own offsets.
- group "email-workers":
- each partition → exactly one
- consumer in the group
- 3 partitions, 2 workers
- → one worker gets 2
- a 4th worker on 3 partitions
- sits idle
- another group (analytics)
- reads the same data independently
Acknowledgements
Acknowledgement timing decides your guarantee. In SQS, a received message becomes invisible for a visibility timeout, say thirty seconds. If the worker doesn't delete it in time, it reappears for someone else. In Kafka, the consumer commits its offset. Acknowledge before doing the work and a crash loses the message: at most once. Acknowledge after, and a crash between work and ack means it's processed twice: at least once.
- queue: message hidden for a
- visibility timeout (e.g. 30 s)
- no ack in time → visible
- again → another worker
- log: commit offset after
- processing
- ack before work → crash loses
- it (at-most-once)
- ack after work → may repeat
Idempotent consumers
Real systems choose at least once, then make the consumer idempotent, as Concepts episode twenty-nine describes. Record each message id in a table with a unique key, in the same transaction as the work. A duplicate hits the unique constraint and is skipped. Or make the effect naturally idempotent: set status to shipped, rather than adding one. Exactly-once delivery across systems is a myth; effectively-once processing is not.
BEGIN; INSERT INTO processed(msg_id) VALUES (77); -- unique key -- duplicate → conflict → skip do the work / write results COMMIT; ack message or make the effect itself idempotent (SET, upsert)
Retries and DLQ
Failures come in two kinds. Transient ones, like a timeout from the email provider, deserve retries with exponential backoff and jitter. Poison messages fail every time, because of bad data or a bug, and without a limit they loop forever or block their partition. After, say, five attempts, move the message to a dead-letter queue and alert. Once the bug is fixed, redrive the messages back to the main queue.
- transient (timeout, 503):
- retry with backoff + jitter
- 1 s, 2 s, 4 s, 8 s ...
- poison (bad data, bug):
- fails every time, blocks
- or loops forever
- → after 5 tries move to DLQ
- DLQ: alert, inspect, fix,
- redrive to the main queue
Ordering
Ordering. Total order across everything means one partition and one consumer, which doesn't scale. You almost never need it. You need per-key order: created, paid, shipped for the same order. Hash the order id to pick the partition and those events stay in sequence. Watch the traps: client retries can reorder messages unless the producer is idempotent, one slow key holds up its whole partition, and adding partitions moves keys to new ones.
- global order = one partition
- = one consumer = no scale
- per-key order: hash(order_id)
- → all events of one order in
- one partition, in order
- traps: retries can reorder;
- one slow key blocks its
- partition; changing the
- partition count moves keys
Durability
Durability comes from replication. Each partition has a leader and two followers on other brokers. With acks set to all and a minimum of two in-sync replicas, the producer gets an ack only when at least two copies exist. If the leader's broker dies, the cluster's controller promotes an in-sync follower within seconds and producers retry. Acks of one is faster, but a crash right after the ack loses the message.
- each partition: 1 leader +
- 2 followers on other brokers
- acks=all + min in-sync = 2:
- ack only when 2 copies exist
- leader dies → controller
- elects an in-sync follower
- (seconds), producers retry
- acks=1: faster, can lose data
Lag and backpressure
The key health metric is consumer lag: how far the committed offset trails the newest message. Steady lag is fine. Growing lag means consumers are slower than producers. Add consumers up to the partition count, process in batches, or fix the slow downstream call. And remember, retention is a deadline. If lag exceeds seven days of retention, messages expire before they're read.
- lag = latest offset −
- committed offset
- growing lag: consumers
- slower than producers
- → add consumers (up to the
- partition count), batch work,
- speed up the slow call
- retention is your deadline:
- lag past 7 days = data loss
Sizing
Sizing, using peak numbers. Twenty thousand one-kilobyte messages a second is twenty megabytes a second. Seven days of retention is about twelve terabytes, thirty-six with three replicas. Now the consumers: if each message takes fifty milliseconds, one thread does twenty a second, so you need a thousand concurrent threads, like a hundred workers with ten threads each. In Kafka that means at least a hundred partitions, so pick a bit more.
peak 20,000 msgs/s × 1 KB = 20 MB/s in retention 7 days: 20 MB/s × 604,800 s ≈ 12 TB × 3 replicas ≈ 36 TB consumer: 50 ms per message → 20 msgs/s per thread 20,000 / 20 = 1,000 threads → 100 workers × 10 threads → ≥ 100 partitions (use 120)
Say it like this
In the interview, justify the queue first: latency, isolation from a flaky dependency, absorbing spikes. Pick the kind: a log like Kafka when you need replay or several readers, a task queue like SQS for simple jobs. Name the partition key and the order it gives. Say at-least-once with an idempotent consumer, retries with backoff and a dead-letter queue, replication factor three with acks all, and that you'd alert on consumer lag.
- why async here (latency,
- isolation, spikes)
- Kafka (replay, fan-out) vs
- SQS (simple task queue)
- key: order_id → per-order
- order
- at-least-once + idempotent
- consumer (msg id table)
- retries + backoff → DLQ
- acks=all, RF 3
- watch lag; size partitions