Start practising

SYSTEM DESIGN · CONCEPTS · DISTRIBUTED SYSTEMS · EPISODE 28

Message Queues and Pub-Sub

Why services hand work to a queue instead of calling each other directly: decoupling, buffering spikes and retries. Work queues vs pub-sub, Kafka-style logs with partitions, offsets and consumer groups, delivery guarantees (at-most-once, at-least-once, effectively once), ordering, and dead-letter queues.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
1 / 12

Press play to hear the narration, or step through at your own pace. Every picture here is the same one used in the YouTube video.

Notes

The problem

The direct version: the order service calls email, inventory and analytics one after another. The user waits for all of them. If analytics is down, do orders fail? And on a sale day, a spike hits every downstream service at the same moment. The services are tightly coupled.

Synchronous fan-out
  • order request calls 3 services
  • latency = sum of the slow ones
  • availability = product of all
  • analytics down → orders fail?
  • Black Friday spike → everything
  • downstream melts at once

The queue

With a queue, the order service writes one message, 'order placed', to a durable queue and returns immediately. Consumers pick it up when they can. The producer doesn't know or wait for them. A spike just makes the queue longer. And if a consumer is down, its messages wait for it instead of being lost.

What a queue gives you
  • decoupling: producer does not
  • know or wait for consumers
  • buffering: a spike fills the
  • queue, workers drain it at
  • their own pace
  • reliability: messages wait
  • while a consumer is down

Queue vs pub-sub

Two shapes. A work queue hands each message to exactly one worker; add workers to go faster. That's SQS or a RabbitMQ queue, perfect for 'resize this image'. Pub-sub is a topic: every subscribing service gets its own copy. Email, inventory and analytics all need to see 'order placed', so that's pub-sub.

Two shapes
  • work queue (point-to-point):
  • each message goes to ONE
  • worker; add workers to
  • scale (SQS, RabbitMQ)
  • pub-sub (topic):
  • each message goes to EVERY
  • subscriber group
  • (Kafka, SNS, Pub/Sub)

Kafka

Kafka works differently from a classic queue. A topic is an append-only log. Messages get an offset and are kept for days, not deleted when read. Each consumer group just remembers its own offset: email has processed up to four, analytics up to two. A new service can even replay from the start.

Kafka: a log, not a queue
topic "orders", partition 0:

offset: 0    1    2    3    4
        o17  o18  o19  o20  o21

messages are appended and kept
(days), not deleted on read

each consumer group tracks its
own offset: email at 4,
analytics at 2

Partitions

To scale, a topic is split into partitions, and a message's key, say the order id, picks its partition. Within a consumer group each partition is read by one worker, so three partitions allow three parallel workers. Order is guaranteed within a partition, not across the whole topic. So put everything that must stay in order under the same key.

Partitions and consumer groups
topic "orders": 3 partitions
key = order_id → partition

group "email" with 3 workers:
  worker 1 ← partition 0
  worker 2 ← partition 1
  worker 3 ← partition 2

order is kept WITHIN a partition
max parallelism = partitions

Guarantees

Delivery guarantees. At-most-once: acknowledge before processing, and a crash loses the message. At-least-once: acknowledge after processing, and a crash means it's delivered again. That's the normal default, so consumers will see duplicates. 'Exactly-once' in practice means at-least-once plus a consumer that safely ignores repeats, which is episode twenty-nine.

Delivery guarantees
  • at-most-once: ack before
  • processing → crash loses it
  • at-least-once: ack after
  • processing → crash means
  • it is delivered again
  • (the usual default)
  • exactly-once: really
  • "effectively once" =
  • at-least-once + idempotent
  • consumer (episode 29)

Dead letters

Some messages always fail: a bad email address, a bug for one weird order. Retried forever, that poison message blocks everything behind it. So retry a few times with growing delays, then move it to a dead-letter queue. Alert when the dead-letter queue grows, fix the bug, and replay the messages.

Retries and dead letters
  • a message that always fails
  • ("poison") would block the
  • queue forever
  • retry with backoff, N times
  • then move it to a
  • dead-letter queue (DLQ)
  • alert on DLQ size,
  • fix, replay

Lag

The one metric to watch is consumer lag: how far a group's position is behind the newest message. Growing lag means consumers can't keep up, so add workers, up to the number of partitions. A queue absorbs a spike for a while; it doesn't make a permanently overloaded system fast.

Watch the lag
  • consumer lag = newest offset
  • minus the group’s offset
  • growing lag → consumers too
  • slow: add workers (up to the
  • number of partitions)
  • backpressure: a queue hides
  • overload for a while,
  • not forever

Decision

When to use one: when the user doesn't need the result right now, like emails, thumbnails and analytics; when traffic is spiky; and when several services react to the same event. Keep it synchronous when the user is waiting for the answer, like authorising a payment or logging in.

When to use one
  • use a queue when:
  • • the user does not need the
  • result right now (email,
  • thumbnails, analytics)
  • • bursts are spiky
  • • several services react to
  • one event
  • keep it sync when the user
  • needs the answer (payment
  • authorisation, login)

Say it like this

In the interview: publish an event to a topic, partitioned by the entity id so related events stay in order. Assume at-least-once delivery and make consumers idempotent. Retry with backoff and send failures to a dead-letter queue. And monitor consumer lag. That's a complete answer in one breath.

In the interview
  • "How do you handle the slow
  • side effects?"
  • publish an event to a topic,
  • partition by entity id,
  • at-least-once delivery,
  • idempotent consumers,
  • retries + DLQ,
  • monitor consumer lag

Interview questions

Why put a message queue between services?

It decouples producer and consumers so the producer does not wait for them, buffers traffic spikes so consumers process at their own pace, and keeps messages durable while a consumer is down.

What is the difference between a work queue and pub-sub?

In a work queue each message is delivered to one of the competing workers, which scales processing. In publish-subscribe each subscriber (or consumer group) receives its own copy of every message.

How is Kafka different from a traditional queue?

Kafka stores messages in an append-only, partitioned log and retains them for a configured period. Consumers track their own offsets per consumer group, so many groups can read the same data independently and replay it.

How does Kafka preserve ordering?

Ordering is guaranteed only within a partition. Messages with the same key go to the same partition, so choosing the entity id as the key keeps all events for that entity in order.

What delivery guarantee do most queues give and what does it imply?

At-least-once: a message is acknowledged after processing, so a crash before the acknowledgement causes redelivery. Consumers must be idempotent to tolerate duplicates.

What is a dead-letter queue?

A separate queue where messages are moved after failing processing a set number of times, so a poison message does not block the main queue. Teams alert on it, fix the cause and replay the messages.

What is consumer lag and how do you reduce it?

The gap between the newest message in a partition and the consumer group’s committed offset. Reduce it by optimising consumers or adding consumers, up to the number of partitions.

When should you not use a queue?

When the caller needs the result immediately to continue, such as authorising a payment or validating a login, a synchronous call is simpler and gives a direct error.