Start practising

SYSTEM DESIGN · CONCEPTS · SERVICES AND HOW THEY TALK · EPISODE 9

Real-Time: WebSockets, Long Polling and SSE

How a server pushes new data to a client: short polling, long polling, server-sent events and WebSockets, how chat apps route a message to the right connection, scaling millions of open connections, and how to pick one in an interview.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
1 / 10

Press play to hear the narration, or step through at your own pace. Every picture here is the same one used in the YouTube video.

Notes

Polling

The simplest approach is polling: ask every few seconds, 'anything new?' It works, but with a million users that's hundreds of thousands of requests a second, almost all answered 'nothing', and a message can still wait seconds to show up. Fine for a dashboard that refreshes. Bad for chat.

Short polling
  • every 3 s: GET /messages?since=…
  • 1M users × 1 request / 3 s
  • = 330,000 requests per second
  • almost all answer "nothing"
  • and a message can wait
  • up to 3 s to appear
  • fine for: dashboards, slow data

Long polling

Long polling is smarter. The client asks, and the server simply doesn't answer until there's something to say, or about thirty seconds pass. When the answer arrives, the client asks again right away. Delivery becomes nearly instant using plain HTTP, but every message still costs a full request.

Long polling
  • client asks, server does NOT
  • answer until data arrives
  • (or a timeout, ~30 s)
  • client immediately asks again
  • + near-instant delivery
  • + plain HTTP, works everywhere
  • − a request per message
  • − holding requests costs memory

SSE

Server-sent events keep one HTTP response open, and the server keeps writing events into it: score one nil, score two nil. It's one direction only, server to client, it's plain HTTP, and browsers reconnect automatically. Perfect for live scores, notifications and streaming text.

Server-sent events
GET /live  (Accept: text/event-stream)

data: {"score":"1-0"}

data: {"score":"2-0"}

server → client only
plain HTTP, auto-reconnect
good for: feeds, live scores,
notifications, AI token streams

WebSockets

WebSockets go further. The connection starts as HTTP, then upgrades into a persistent two-way channel. Both sides can send whenever they like, with tiny frames and no headers per message. This is what chat, multiplayer games and collaborative editors use. The cost: each connection is long-lived state the server must hold.

WebSockets
  • starts as HTTP, then "Upgrade"
  • to a persistent TCP connection
  • both sides send anytime
  • tiny frames, no headers each time
  • good for: chat, multiplayer games,
  • collaborative editing, trading
  • cost: stateful, long-lived
  • connections to manage

Routing

With many chat servers, Alice may be connected to server one and Bob to server two. So each server records who is connected where, often in Redis: Bob is on server two. To deliver, server one looks Bob up and forwards the message, directly or through a pub-sub channel. If Bob is offline, store it and send a push notification.

Connection registry
on connect:
  SET conn:bob = server-2

to deliver:
  look up conn:bob
  forward to server-2 (or pub-sub
  channel per server)

offline? store the message,
send a push notification

Scaling

A single server with an event loop can hold hundreds of thousands of mostly idle connections. The load balancer has to support long-lived connections, heartbeats detect dead ones, and when a server restarts, clients reconnect with backoff and resume from the last message id they saw, so nothing is lost.

Scaling connections
  • one server: ~100k–1M idle
  • sockets with an event loop
  • load balancer must support
  • long-lived connections
  • deploys drop connections:
  • clients reconnect with backoff
  • and resume from last seen id
  • heartbeats detect dead ones

Decision

How to choose? Data that changes every minute: just poll. A one-way stream like notifications or a live feed: server-sent events. Two-way and low latency, like chat, games or collaborative editing: WebSockets. And when the app is in the background, none of these work, so you use mobile push notifications.

Which one?
  • updates every minute → polling
  • server → client stream,
  • notifications, feeds → SSE
  • two-way, low latency,
  • chat, games, editors → WebSockets
  • need it to work anywhere,
  • simple infra → long polling
  • mobile apps in background:
  • push notifications (APNs/FCM)

Say it like this

The interview answer for chat. Each online client holds a WebSocket to a gateway server. A registry maps each user to their gateway. Save the message first, then route it through pub-sub to the right gateway. Offline users get a push notification, and reconnecting clients resume from the last message id.

In the interview
  • "How does Bob get it instantly?"
  • WebSocket per online client
  • gateway servers hold sockets
  • registry: user → gateway
  • route via pub-sub
  • store message first, then deliver
  • offline → push notification
  • reconnect + resume from last id

Interview questions

What is the difference between polling and long polling?

Polling sends a request at a fixed interval and usually gets an empty answer. Long polling holds each request open until new data arrives or a timeout, then the client re-requests, giving near-instant delivery with fewer empty responses.

When would you use server-sent events instead of WebSockets?

When data flows only from server to client, such as notifications, live scores or streamed text. SSE is plain HTTP, simpler to operate and reconnects automatically. Use WebSockets when the client also sends frequent messages.

How does a WebSocket connection start?

As an HTTP request with an Upgrade header. After the server agrees, the same TCP connection becomes a persistent, full-duplex channel for small frames in both directions.

How does a chat system deliver a message when sender and receiver are on different servers?

Each gateway records which users are connected to it in a shared registry. The sender's server looks up the receiver's gateway and forwards the message directly or via a pub-sub channel; if offline, the message is stored and a push notification is sent.

What happens to WebSocket clients when a server is deployed or crashes?

Their connections drop. Clients reconnect with exponential backoff, possibly to another server, and request messages after the last id they received so nothing is missed.

How many connections can one server hold?

With an event-driven server and tuned limits, hundreds of thousands to around a million mostly idle connections; the real limits are memory per connection, file descriptors and message throughput.

How do mobile apps receive messages in the background?

Through platform push services (Apple Push Notification service, Firebase Cloud Messaging), because the OS closes long-lived connections for background apps.

Why are WebSockets harder to scale than plain HTTP?

Connections are long-lived and stateful, so load balancers must support them, servers need routing to find a user's connection, and deploys or failures force mass reconnections.