Real-Time Updates: Pushing Changes to Millions of Open Screens

The signals

Listen for phrases like live, as it happens, or without refreshing. Chat, typing indicators, a courier on a map, live scores, stock prices, shared editing. In all of them the interesting direction is server to client, and the target is usually under a second. When you hear those, reach for this pattern.

The naive design

The naive design is polling: every client asks every couple of seconds whether anything changed. With a million open screens that is half a million requests a second, and if only five thousand things actually change per second, ninety-nine percent of that work returns nothing. Worse, you are still up to two seconds stale. Polling faster just burns more servers.

1M open screens
poll every 2 s
→ 500,000 requests/s

real changes: ~5,000/s
→ 99% of requests empty

and still up to 2 s stale
faster polling = more load

Move 1: long polling

The simplest upgrade is long polling. The client asks, and the server does not answer until there is something new or about thirty seconds pass. Then the client asks again straight away. Delivery is almost instant and it is plain HTTP, so it works through any proxy. The cost is a new request per event, and held requests that tie up server threads unless the server is asynchronous.

Move 2: SSE

Second move, Server-Sent Events. The client opens one HTTP request and the server keeps the response open, writing an event whenever something happens. It is one-way, server to client, which is exactly what a feed, a score or a notification badge needs. Browsers reconnect automatically and send the last event ID they saw, so the server can resume. Writes go up as ordinary POST requests.

Move 3: WebSockets

Third move, WebSockets. The connection starts as HTTP, upgrades, and becomes a two-way pipe with only a few bytes of overhead per message. Use it when the client also sends frequent small messages: chat, typing indicators, games, collaborative editing. The price is that you own reconnects, acknowledgements and heartbeats, and the connection is stateful, which makes load balancing harder. See Concepts episode 9.

Choosing

How to choose. If a few updates a minute are fine, plain polling is honestly acceptable and cheapest to build. If updates flow one way, pick SSE. If the client chatters back constantly, pick WebSockets. Long polling is the fallback when a network blocks the others. Say it in that order, simplest first, and the interviewer sees judgement rather than buzzwords.

need          poll  long  SSE  WS
updates/min    ok    ok    ok   ok
sub-second     no    ok    ok   ok
client sends   -     -     POST yes
often
through any    yes   yes   most most
proxy

default: SSE for one-way feeds,
WebSockets for two-way chatter

Connection tier

Now scale it. Open sockets are state, so put them on their own tier of connection servers that do nothing but hold connections and relay messages. Your app servers stay stateless and easy to deploy. A well-tuned box can hold a few hundred thousand idle connections, so ten million online users is roughly twenty of them. Use a layer four balancer with long idle timeouts, and send heartbeats so dead connections are noticed.

Routing events

The hard question: user forty-two is connected to one of twenty boxes. Which one? The cleanest answer is pub-sub. When the user connects, that connection server subscribes to the channel user forty-two. The app publishes the event to that channel, and pub-sub delivers it to whichever server is subscribed. The app never needs to know where anyone is connected.

The alternative is a connection registry. Each connection server records user to server in Redis with a short expiry, refreshed by heartbeats. The app looks up the server and calls it directly. That avoids pub-sub traffic, but entries go stale after a crash, so every send needs a timeout and a fallback. Both are fine answers; name the trade-off.

Fan-out

Fan-out is the next trap. A goal in a final might go to two million screens. Never loop over two million users in the app. Publish once to the room's channel, let each connection server fan out to its own local viewers, and batch updates every couple of hundred milliseconds. For floods like live comments, sample them. Nobody can read five thousand comments a second anyway.

Reconnects

Phones lose signal all the time, so treat the push as a fast hint and the durable store as the truth. Write each event with a sequence number before pushing it. The client remembers the last number it saw. On reconnect it says, give me everything after one hundred eighteen, and the server replays the gap. That gives at-least-once delivery; the client drops duplicates by event ID.

Failure modes

The failure modes interviewers probe. Deploying a connection server drops half a million sockets, so drain them gradually. After an outage every client reconnects at once, so use exponential backoff with random jitter. A slow phone on a bad network can fill your send buffer, so cap it and skip stale updates. And heartbeats catch zombie connections that never closed cleanly.

Trade-offs

The trade-offs to say out loud. Push gives freshness but costs stateful infrastructure that polling does not need. SSE is simpler, WebSockets are more capable. Pub-sub routing spends broadcast traffic, a registry risks stale entries. You promise at-least-once delivery with deduplication, not exactly once, and ordering per conversation or per document, not across the whole system.

Where it shows up

Where you will use this: WhatsApp and Slack for messages and presence, live comments for huge fan-out, Google Docs for edits in both directions, Uber and food delivery for moving dots on a map, a stock exchange for price ticks, a leaderboard for rank changes, and the notification system for unread badges. Each of those has its own problem episode.

Say it like this

Here is how to say it. Clients keep a WebSocket, or SSE if it is one-way, to a dedicated connection tier behind a layer four balancer. The app publishes events to a per-user or per-room channel, and the server holding that socket pushes it. Events are stored with sequence numbers so a reconnecting client catches up. Big rooms fan out per server and batch.