Notes
The problem
Without pub-sub, the order service calls email, analytics, search and loyalty one by one. Every new consumer means changing and redeploying the publisher, and any slow consumer slows checkout. What you want is to publish one event, order placed, and let any team subscribe later without the publisher knowing or caring.
- order service calls email,
- analytics, search, loyalty...
- new consumer = change and
- redeploy the publisher
- one slow consumer slows
- checkout
- want: publish once, anyone
- can subscribe later
How it works
The publisher sends one event to a topic. Each subscription, one per consuming service, receives its own copy of every event. Inside one subscription, several workers can share the load like a normal queue. So pub-sub is really queue semantics per subscriber, multiplied. The publisher only knows the topic name; adding a subscriber is a configuration change, not a code change.
- topic: a named stream of
- events (order-placed)
- subscription: one consumer
- type's view of the topic
- every subscription gets every
- event; inside a subscription,
- workers share the load
- publisher knows only the topic
Variants
There are two ways to build it. Copy per subscriber: a topic pushes each event into a separate durable queue per subscriber, like SNS fanning out to several SQS queues. Each queue has its own retries and dead letters. Or a shared log: Kafka stores each event once, and every subscribing service is a consumer group with its own offset. Storage doesn't multiply, and a new subscriber can replay history from the start.
- copy per subscriber:
- SNS topic → one SQS queue
- per subscriber (fan-out)
- Google Pub/Sub subscriptions
- shared log, offset per
- subscriber:
- Kafka topic + one consumer
- group per service
- one stored copy, many readers
- replay = move the offset
Push vs pull
Delivery is push or pull. With push, the broker calls your HTTP endpoint for each event, which is simple for serverless functions, but the broker must back off when you're slow or down. With pull, the subscriber fetches batches at its own pace. That gives natural backpressure and higher throughput through batching, at the cost of running workers all the time. High-volume service subscribers usually pull.
- push: broker calls your HTTP
- endpoint (webhook style)
- + simple for serverless
- - broker must back off when
- you are slow or down
- pull: subscriber fetches
- batches at its own pace
- + natural backpressure
- + batching, high throughput
- - needs always-on workers
Filtering
Not every subscriber wants every event. Filter at the subscription, on attributes like event type or country, so the email service in India only receives Indian orders. Or split events into separate topics by type. And because many teams depend on the same event, version its schema in a registry and evolve it safely: add optional fields, never rename or remove one, or a subscriber you've never heard of breaks.
- subscription filter on
- attributes: type=refund,
- country=IN
- or separate topics per event
- type (coarse but simple)
- schema registry: versioned
- event schemas (Avro/Proto);
- add fields, never rename
- or remove
Failure isolation
Isolation is the point. If analytics is down for two hours, its events pile up in its own queue or behind its own offset, while email and notifications carry on. When it recovers, it catches up. Set retention longer than the longest outage you'll tolerate. Be careful with ephemeral pub-sub like Redis publish and subscribe: if a subscriber isn't connected at that moment, the message is simply gone.
- durable subscriptions: its
- events wait in its own queue
- or behind its own offset
- others keep going
- set retention ≥ the longest
- outage you will tolerate
- ephemeral pub-sub (Redis
- PUBLISH): offline = missed
Fan-out to devices
Fan-out to devices adds a twist: users hold WebSocket connections spread across many gateway servers, each holding a hundred thousand or more. To deliver an event, you need to know which gateway holds which user. Keep a registry, updated on connect, or let each gateway subscribe to the channels its connected users care about. Users who aren't connected get a mobile push notification instead.
- millions of open WebSockets
- spread over gateway servers
- (~100k-1M connections each)
- registry: user → gateway
- (Redis, updated on connect)
- or channel subscriptions:
- gateway subscribes to the
- channels its users watch
- offline → push notification
Write vs read
For feeds, fan-out has a famous trade-off. Fan-out on write copies each new post into every follower's feed cache, so reading a feed is one lookup. But a celebrity with fifty million followers turns one post into fifty million writes. Fan-out on read builds the feed at read time from the accounts you follow: cheap writes, slow reads. The standard answer is hybrid: push posts from normal users, and merge in celebrity posts at read time.
- on write (push): copy post
- into each follower's feed
- + reads are one lookup
- - 50M followers = 50M writes
- on read (pull): build feed
- from followed accounts at
- read time
- + cheap writes, - slow reads
- hybrid: push normal users,
- pull celebrities
Guarantees
Guarantees carry over from queues, once per subscriber. Each one gets at-least-once delivery, so every handler must be idempotent, keyed on the event id. Ordering is only per key, and two subscribers will see the same event at different moments, so never assume search and email agree at any instant. Put a version number on events about an entity, so a late or repeated older event can be ignored.
- each subscriber sees at-least-
- once delivery → idempotent
- handlers (event id)
- order only per key/partition;
- subscribers may see events
- at different times
- carry a version or timestamp
- so late or repeated events
- are ignored
Sizing
Always size the amplification. Two thousand orders a second with six subscribers is twelve thousand deliveries a second, easy. Five hundred chat messages a second into rooms with two hundred online members each is a hundred thousand pushes a second, which needs a real gateway fleet. One celebrity post to fifty million followers, ten percent online, is five million pushes; at half a million a second, it takes ten seconds to spread, and that's acceptable.
2,000 orders/s × 6 subscribers = 12,000 deliveries/s chat: 500 msgs/s into rooms avg 200 members online = 100,000 pushes/s one celebrity post: 50M followers, 10% online = 5M pushes; at 500k/s spread over ~10 s
Scaling
To scale, partition topics so each subscriber can run many workers. For device fan-out, go in tiers: send one copy of the message to each gateway that has interested users, and let each gateway push to its own local connections, rather than one service sending millions of individual messages. Shard very large channels into sub-channels, and put a cap on how much work any single event can trigger.
- partition the topic so each
- subscriber scales its workers
- shard huge channels:
- channel → N sub-channels
- tiered fan-out: one message
- to 100 gateways, each pushes
- to its local sockets
- cap fan-out work per event
Say it like this
In the interview, say the publisher emits a business event to a topic and doesn't know its subscribers. Each subscriber has its own durable queue or consumer group, with retries, a dead-letter queue and idempotent handlers. For devices, describe the gateway registry and tiered push. For feeds, the hybrid push and pull. And always compute deliveries per second as events times audience.
- "Order service publishes
- order-placed; email, search,
- analytics subscribe"
- durable subscription each
- (own queue or consumer group)
- idempotent handlers, DLQ each
- devices: gateway registry
- feeds: hybrid push/pull
- size: events × audience