SYSTEM DESIGN · DEEP DIVE · 1M REQUESTS PER SECOND · PART 2 OF 6

Handling 1 Million Requests per Second, Part 2: The Network Edge and Load Balancing

How a million requests a second reach the app servers: what a new TCP and TLS connection costs, keep-alive, HTTP/2 and HTTP/3, the load balancing tiers (GeoDNS, anycast, CDN, L4, L7) and the capacity of each, balancing algorithms, the long-lived connection trap, ephemeral port exhaustion, health checks and draining, and multiple regions.

Part 2 of 6 in How to Handle 1 Million Requests per Second, a deep dive in 6 videos.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
1 / 20

Press play to hear the narration, or step through at your own pace.

Notes

The path of a request

Here is the path. The client first asks DNS, which answers with the address of a nearby healthy region. That address is a virtual IP owned by a layer-four load balancer, which forwards packets for each connection to one of many layer-seven proxies. The proxy decrypts TLS, understands HTTP, and sends each request to an app server. Coarse decisions happen early and cheaply, fine decisions late, where they cost more.

Connections cost

First, connections are expensive. A new connection costs the user a TCP handshake and a TLS handshake, two round trips before the first byte of the request. It costs the server real CPU for the public-key math: roughly half a millisecond with an RSA certificate, about a tenth with ECDSA. If every request opened a new connection, the handshakes alone would need hundreds of cores. So the first rule of the edge: reuse connections.

Price of a new connection
TCP handshake       1 round trip
TLS 1.3             1 round trip
  (TLS 1.2: 2 round trips)

server CPU per full handshake:
  RSA-2048 cert   ~0.5 ms
  ECDSA P-256     ~0.1 ms

1,000,000 new TLS conns/s:
  RSA ≈ 500 cores, ECDSA ≈ 100
just to say hello

Keep-alive

With keep-alive, one connection carries many requests. If each connection serves a hundred requests, a million requests a second needs only ten thousand handshakes a second, about one core with ECDSA. TLS session resumption makes reconnects cheaper still, because a returning client skips the expensive key work. The catch with HTTP one point one is that each connection carries one request at a time, so browsers open several connections per host.

Keep-alive math
100 requests per connection
→ 10,000 handshakes/s, not 1M
→ ECDSA: about 1 core

also: TLS session resumption
  (tickets) skips the costly
  key work on reconnect

HTTP/1.1 keep-alive still sends
one request at a time per conn

HTTP/2 and HTTP/3

HTTP two goes further. Many requests share one connection as parallel streams, and header compression means the seven hundred bytes of headers we measured in part one shrink to a few bytes after the first request. Its weakness: one lost packet pauses every stream, because TCP delivers bytes in order. HTTP three runs QUIC over UDP, so a loss stalls only one stream, setup is faster, and connections survive a phone switching networks. It costs more CPU per byte.

HTTP/2 and HTTP/3
  • HTTP/2: many requests share one
  • connection as streams
  • HPACK: repeated headers shrink
  • from ~700 B to a few bytes
  • one lost packet stalls all
  • streams (TCP ordering)
  • HTTP/3 (QUIC over UDP):
  • 1 round trip setup, 0 on resume
  • loss stalls only one stream
  • survives Wi-Fi to 4G switch
  • costs more CPU per byte

Tier 1: DNS

Tier one is DNS. A geo-aware DNS service answers each lookup with the address of the nearest healthy region. It is almost free, but it is coarse. Answers are cached with a time to live of maybe thirty to sixty seconds, and many resolvers and apps keep them longer than asked. So failover through DNS takes minutes, and DNS has no idea how loaded each server is. Use it to pick a region, never to balance individual requests.

GeoDNS
  • answers with the region that is
  • nearest and healthy
  • TTL 30–60 s, but resolvers and
  • apps cache longer than asked
  • failover takes minutes, not
  • seconds
  • it cannot see server load
  • good for: picking a region
  • bad for: balancing requests

Anycast and the CDN

Before our own data center sits a CDN using anycast: the same IP address is announced from hundreds of edge sites, and the internet’s routing delivers each user to the nearest one. The edge finishes the TLS handshake ten milliseconds from the user instead of a hundred and fifty. It then sends requests to our origin over a few warm, reused connections. It can even cache responses for a second or two, which part four uses heavily, and it soaks up floods.

Anycast + CDN
  • the same IP is announced from
  • hundreds of edge sites (BGP)
  • routers deliver to the nearest
  • the edge finishes TLS close to
  • the user: short round trips
  • keeps warm, pooled connections
  • back to our origin
  • can cache GET responses for a
  • second or two (part 4)
  • absorbs floods before origin

Tier 2: L4

Tier two is the layer-four balancer. It looks only at addresses and ports, never at HTTP. For each connection it hashes the source and destination and picks a proxy, so every packet of that connection lands on the same proxy. With consistent hashing, losing one balancer box does not reshuffle every connection. Routers spread packets across the balancer boxes, and with direct server return the responses, which are the big part, can skip the balancer entirely.

How much does layer four cost? A small request and response with acknowledgements is around five packets, so a million requests a second is about five million packets a second. Balancers built on kernel bypass, such as XDP or DPDK, forward millions of packets per core because they never build a full TCP connection. Four boxes, any two of which can carry the peak, is plenty. Cheap per packet, but blind to anything inside the request.

Layer 4 balancing
  • sees only IPs and ports
  • hashes each connection to one
  • proxy, so all its packets
  • go to the same place
  • consistent hashing: losing one
  • LB box does not reshuffle
  • every connection
  • routers spread packets across
  • LB boxes with ECMP
  • direct server return: replies
  • can skip the balancer
L4 capacity
~5 packets per small request:
  request, response, ACKs
1,000,000 req/s ≈ 5M packets/s

kernel-bypass balancers (XDP,
  DPDK) move millions of
  packets/s per core

plan: 4 boxes, any 2 can carry
  the peak
cheap per packet, blind to HTTP

Tier 3: L7

Tier three is the layer-seven proxy, such as Envoy, NGINX or HAProxy. It finishes TLS, parses HTTP, and makes a decision for every request: which service by host and path, which server, whether to retry, whether the caller is over its rate limit. It adds request IDs and records metrics. All that intelligence costs CPU on every request, so unlike DNS or layer four, this tier must be sized carefully.

Let’s size it with an assumed figure: one proxy core with TLS and keep-alive handles about twenty thousand simple requests a second. Measure your own. A million needs fifty cores, a hundred at fifty percent, so thirteen eight-core boxes plus two spares. Connections matter too. Two million open client connections across fifteen boxes is about a hundred and thirty thousand each, around four gigabytes of buffers and the same number of file descriptors.

Layer 7 proxy
  • terminates TLS, parses HTTP
  • routes by host, path, header
  • balances per request, not per
  • connection
  • retries, timeouts, rate limits
  • adds request IDs, metrics
  • Envoy, NGINX, HAProxy
  • every request costs CPU here:
  • this tier must be sized
L7 sizing (assumed numbers)
one proxy core with TLS and
  keep-alive: ~20,000 req/s
1,000,000 ÷ 20,000 = 50 cores
at 50% target → 100 cores
→ 13 boxes of 8 cores + 2 spare

client connections: ~2,000,000
  ÷ 15 ≈ 133,000 per box
  × ~30 KB buffers ≈ 4 GB RAM
  and 133,000 file descriptors

Balancing algorithms

How should a proxy choose a server? Round robin gives each the same count but ignores that one may be slow. Least outstanding requests is smarter, but fifteen proxies with slightly stale views can all send their next burst to the same idle server. The best everyday answer is the power of two choices: pick two servers at random, send to the less busy one. It gets almost the full benefit with no herding.

Choosing a server
  • round robin: even count, blind
  • to slow servers
  • least outstanding requests:
  • good, but 15 proxies with
  • stale views all pick the
  • same "best" server
  • power of two choices: pick 2 at
  • random, take the less busy
  • → near-optimal, no herding
  • weights for bigger machines

The long connection trap

Here is a trap. HTTP two and gRPC connections can live for hours, and a layer-four balancer only balances connections. When you add servers during a spike, the new ones receive almost nothing, because every existing connection stays where it was. The fix: balance requests at layer seven, and give connections a maximum age, say five minutes plus some random jitter, after which the server politely asks the client to reconnect.

Pinned connections
  • HTTP/2 and gRPC connections
  • can live for hours
  • L4 balances connections, not
  • requests
  • scale out: new servers get no
  • traffic, old ones stay hot
  • fix: balance per request at L7
  • cap connection age, e.g. 5 min
  • with jitter, then GOAWAY
  • cap requests per connection

Ephemeral ports

Behind the proxy there is another limit people forget. Every connection from a proxy to one app server address uses a local port, and Linux by default offers about twenty-eight thousand of them. A closed connection holds its port in TIME WAIT for sixty seconds. So without reuse you can open only about four hundred and seventy new connections a second to each server. Opening one per request at forty thousand a second exhausts the ports almost immediately.

The fix is mostly keep-alive again: each proxy keeps a pool of warm connections to each app server, say sixty-four, and reuses them indefinitely. Then widen the local port range, enable TCP time-wait reuse for outgoing connections, and if needed give servers more than one address. Alert on the TIME WAIT count and on connect errors. The same limit also hits your load generators in part five, and any service calling Redis or the database.

Port exhaustion
proxy → app: each connection to
  one ip:port uses a local port
default range 32768–60999
  = 28,232 ports
closed conns sit in TIME_WAIT
  for 60 s on Linux
→ ~470 new conns/s per pair

one new conn per request at
  40,000 req/s: out of ports
  in under a second
Fixing port exhaustion
  • keep-alive pools: e.g. 64 warm
  • connections per proxy per
  • app server, reused forever
  • widen the range:
  • net.ipv4.ip_local_port_range
  • = 1024 65535
  • tcp_tw_reuse = 1 for outgoing
  • several IPs or ports per server
  • alert on TIME_WAIT count and
  • connect errors

Health and draining

Health handling matters more at this scale. Proxies check each server actively, every couple of seconds, and also passively: five errors in a row and the server is ejected for thirty seconds. On a deploy, a server drains: it stops getting new requests and finishes the ones in flight. A new server gets slow start, ramping up over a minute while its JIT compiler and caches warm. And never let health checks eject more than about a third of the fleet at once.

Health, draining, slow start
  • active check: GET /health every
  • 2 s, 3 failures → out
  • passive: 5 errors in a row →
  • eject for 30 s
  • draining on deploy: stop new
  • requests, finish in-flight
  • ones for up to 30 s
  • slow start: ramp a new server
  • over 60 s (cold JIT, caches)
  • never eject more than ~30%

Reject early

At a million requests a second, some of that traffic is bots, scrapers and floods. Reject it at the cheapest layer that can recognise it. The CDN edge handles per-IP rules and bot checks. The layer-four tier uses SYN cookies against connection floods. The proxy enforces per-key rate limits and answers four twenty-nine with a retry-after header. The deeper a bad request travels, the more it costs to refuse it.

Cost per rejected request
  • CDN edge: per IP rules, bot
  • checks, cheapest
  • L4: SYN cookies against SYN
  • floods
  • L7: per API key rate limits,
  • 429 with Retry-After
  • app: business rules only
  • the deeper a bad request gets,
  • the more it costs to refuse

More than one region

One region is a single point of failure, so run three, all active. Normally each serves about three hundred and thirty thousand requests a second. Size each for five hundred thousand, so if one region fails, the other two carry the full million. That means paying for one and a half times the capacity you use on a normal day, which is the price of surviving a region outage. The price data must be replicated to every region, which part four covers.

Regions
3 regions, all active
normal: ~333,000 req/s each
size each for 500,000

lose one region:
  2 × 500,000 = 1,000,000 ✓

cost: 1.5x the capacity
state: prices replicated to
  every region (part 4)

Say it like this

In the interview: GeoDNS picks one of three active regions. An anycast CDN finishes TLS near the user and reuses connections to us. Layer-four balancers spread connections over about fifteen layer-seven proxies. The proxies balance each request with power of two choices, over pooled keep-alive connections, so we never run out of ports. Connections get a maximum age so new servers receive traffic. Then mention health checks and draining.

In the interview
  • "GeoDNS picks one of 3 regions.
  • Anycast CDN ends TLS near the
  • user. L4 spreads connections
  • over ~15 L7 proxies, which
  • balance each request with
  • power of two choices over
  • pooled keep-alive connections.
  • Connections get a max age."

Interview questions

What is the difference between L4 and L7 load balancing?

L4 forwards packets by IP address and port without reading the request, so it is cheap and handles millions of packets per second. L7 terminates TLS and parses HTTP, so it can route by path or header and balance each request, but it costs CPU on every request.

Why is keep-alive essential at 1 million requests per second?

Each new connection costs TCP and TLS round trips and public-key CPU work. Reusing a connection for about 100 requests turns a million handshakes per second into ten thousand, saving hundreds of cores and user latency.

Why is DNS not enough to balance load?

DNS answers are cached for at least the TTL and often longer, and DNS cannot see server load. It is good for choosing a region, but failover takes minutes and it cannot balance individual requests.

What is ephemeral port exhaustion and how do you avoid it?

Each outgoing connection to one destination uses a local port, about 28,000 by default, and closed ones wait 60 seconds in TIME_WAIT. Opening a connection per request runs out quickly. Use pooled keep-alive connections, widen the port range and enable tcp_tw_reuse.

Why do new servers get no traffic with long-lived HTTP/2 or gRPC connections?

Connection-level balancing places each connection once, so existing connections stay on old servers. Balance per request at L7 and set a maximum connection age with jitter so clients reconnect and spread out.

What is the power of two choices?

Pick two servers at random and send the request to the less loaded one. It performs close to always picking the least loaded server, but avoids many balancers herding onto the same server based on stale data.

How do you size the L7 proxy tier?

Measure requests per second per proxy core with TLS and keep-alive, divide the peak by it, then add headroom and spares. Also check connection counts, memory for buffers and file descriptor limits per box.

How do you survive a region failure at 1 million requests per second?

Run several active regions and size each to carry its share of the peak if one fails, for example three regions each sized for 500,000 requests per second, with data replicated to all of them.