Notes
The path of a request
Here is the path. The client first asks DNS, which answers with the address of a nearby healthy region. That address is a virtual IP owned by a layer-four load balancer, which forwards packets for each connection to one of many layer-seven proxies. The proxy decrypts TLS, understands HTTP, and sends each request to an app server. Coarse decisions happen early and cheaply, fine decisions late, where they cost more.
Connections cost
First, connections are expensive. A new connection costs the user a TCP handshake and a TLS handshake, two round trips before the first byte of the request. It costs the server real CPU for the public-key math: roughly half a millisecond with an RSA certificate, about a tenth with ECDSA. If every request opened a new connection, the handshakes alone would need hundreds of cores. So the first rule of the edge: reuse connections.
TCP handshake 1 round trip TLS 1.3 1 round trip (TLS 1.2: 2 round trips) server CPU per full handshake: RSA-2048 cert ~0.5 ms ECDSA P-256 ~0.1 ms 1,000,000 new TLS conns/s: RSA ≈ 500 cores, ECDSA ≈ 100 just to say hello
Keep-alive
With keep-alive, one connection carries many requests. If each connection serves a hundred requests, a million requests a second needs only ten thousand handshakes a second, about one core with ECDSA. TLS session resumption makes reconnects cheaper still, because a returning client skips the expensive key work. The catch with HTTP one point one is that each connection carries one request at a time, so browsers open several connections per host.
100 requests per connection → 10,000 handshakes/s, not 1M → ECDSA: about 1 core also: TLS session resumption (tickets) skips the costly key work on reconnect HTTP/1.1 keep-alive still sends one request at a time per conn
HTTP/2 and HTTP/3
HTTP two goes further. Many requests share one connection as parallel streams, and header compression means the seven hundred bytes of headers we measured in part one shrink to a few bytes after the first request. Its weakness: one lost packet pauses every stream, because TCP delivers bytes in order. HTTP three runs QUIC over UDP, so a loss stalls only one stream, setup is faster, and connections survive a phone switching networks. It costs more CPU per byte.
- HTTP/2: many requests share one
- connection as streams
- HPACK: repeated headers shrink
- from ~700 B to a few bytes
- one lost packet stalls all
- streams (TCP ordering)
- HTTP/3 (QUIC over UDP):
- 1 round trip setup, 0 on resume
- loss stalls only one stream
- survives Wi-Fi to 4G switch
- costs more CPU per byte
Tier 1: DNS
Tier one is DNS. A geo-aware DNS service answers each lookup with the address of the nearest healthy region. It is almost free, but it is coarse. Answers are cached with a time to live of maybe thirty to sixty seconds, and many resolvers and apps keep them longer than asked. So failover through DNS takes minutes, and DNS has no idea how loaded each server is. Use it to pick a region, never to balance individual requests.
- answers with the region that is
- nearest and healthy
- TTL 30–60 s, but resolvers and
- apps cache longer than asked
- failover takes minutes, not
- seconds
- it cannot see server load
- good for: picking a region
- bad for: balancing requests
Anycast and the CDN
Before our own data center sits a CDN using anycast: the same IP address is announced from hundreds of edge sites, and the internet’s routing delivers each user to the nearest one. The edge finishes the TLS handshake ten milliseconds from the user instead of a hundred and fifty. It then sends requests to our origin over a few warm, reused connections. It can even cache responses for a second or two, which part four uses heavily, and it soaks up floods.
- the same IP is announced from
- hundreds of edge sites (BGP)
- routers deliver to the nearest
- the edge finishes TLS close to
- the user: short round trips
- keeps warm, pooled connections
- back to our origin
- can cache GET responses for a
- second or two (part 4)
- absorbs floods before origin
Tier 2: L4
Tier two is the layer-four balancer. It looks only at addresses and ports, never at HTTP. For each connection it hashes the source and destination and picks a proxy, so every packet of that connection lands on the same proxy. With consistent hashing, losing one balancer box does not reshuffle every connection. Routers spread packets across the balancer boxes, and with direct server return the responses, which are the big part, can skip the balancer entirely.
How much does layer four cost? A small request and response with acknowledgements is around five packets, so a million requests a second is about five million packets a second. Balancers built on kernel bypass, such as XDP or DPDK, forward millions of packets per core because they never build a full TCP connection. Four boxes, any two of which can carry the peak, is plenty. Cheap per packet, but blind to anything inside the request.
- sees only IPs and ports
- hashes each connection to one
- proxy, so all its packets
- go to the same place
- consistent hashing: losing one
- LB box does not reshuffle
- every connection
- routers spread packets across
- LB boxes with ECMP
- direct server return: replies
- can skip the balancer
~5 packets per small request: request, response, ACKs 1,000,000 req/s ≈ 5M packets/s kernel-bypass balancers (XDP, DPDK) move millions of packets/s per core plan: 4 boxes, any 2 can carry the peak cheap per packet, blind to HTTP
Tier 3: L7
Tier three is the layer-seven proxy, such as Envoy, NGINX or HAProxy. It finishes TLS, parses HTTP, and makes a decision for every request: which service by host and path, which server, whether to retry, whether the caller is over its rate limit. It adds request IDs and records metrics. All that intelligence costs CPU on every request, so unlike DNS or layer four, this tier must be sized carefully.
Let’s size it with an assumed figure: one proxy core with TLS and keep-alive handles about twenty thousand simple requests a second. Measure your own. A million needs fifty cores, a hundred at fifty percent, so thirteen eight-core boxes plus two spares. Connections matter too. Two million open client connections across fifteen boxes is about a hundred and thirty thousand each, around four gigabytes of buffers and the same number of file descriptors.
- terminates TLS, parses HTTP
- routes by host, path, header
- balances per request, not per
- connection
- retries, timeouts, rate limits
- adds request IDs, metrics
- Envoy, NGINX, HAProxy
- every request costs CPU here:
- this tier must be sized
one proxy core with TLS and keep-alive: ~20,000 req/s 1,000,000 ÷ 20,000 = 50 cores at 50% target → 100 cores → 13 boxes of 8 cores + 2 spare client connections: ~2,000,000 ÷ 15 ≈ 133,000 per box × ~30 KB buffers ≈ 4 GB RAM and 133,000 file descriptors
Balancing algorithms
How should a proxy choose a server? Round robin gives each the same count but ignores that one may be slow. Least outstanding requests is smarter, but fifteen proxies with slightly stale views can all send their next burst to the same idle server. The best everyday answer is the power of two choices: pick two servers at random, send to the less busy one. It gets almost the full benefit with no herding.
- round robin: even count, blind
- to slow servers
- least outstanding requests:
- good, but 15 proxies with
- stale views all pick the
- same "best" server
- power of two choices: pick 2 at
- random, take the less busy
- → near-optimal, no herding
- weights for bigger machines
The long connection trap
Here is a trap. HTTP two and gRPC connections can live for hours, and a layer-four balancer only balances connections. When you add servers during a spike, the new ones receive almost nothing, because every existing connection stays where it was. The fix: balance requests at layer seven, and give connections a maximum age, say five minutes plus some random jitter, after which the server politely asks the client to reconnect.
- HTTP/2 and gRPC connections
- can live for hours
- L4 balances connections, not
- requests
- scale out: new servers get no
- traffic, old ones stay hot
- fix: balance per request at L7
- cap connection age, e.g. 5 min
- with jitter, then GOAWAY
- cap requests per connection
Ephemeral ports
Behind the proxy there is another limit people forget. Every connection from a proxy to one app server address uses a local port, and Linux by default offers about twenty-eight thousand of them. A closed connection holds its port in TIME WAIT for sixty seconds. So without reuse you can open only about four hundred and seventy new connections a second to each server. Opening one per request at forty thousand a second exhausts the ports almost immediately.
The fix is mostly keep-alive again: each proxy keeps a pool of warm connections to each app server, say sixty-four, and reuses them indefinitely. Then widen the local port range, enable TCP time-wait reuse for outgoing connections, and if needed give servers more than one address. Alert on the TIME WAIT count and on connect errors. The same limit also hits your load generators in part five, and any service calling Redis or the database.
proxy → app: each connection to one ip:port uses a local port default range 32768–60999 = 28,232 ports closed conns sit in TIME_WAIT for 60 s on Linux → ~470 new conns/s per pair one new conn per request at 40,000 req/s: out of ports in under a second
- keep-alive pools: e.g. 64 warm
- connections per proxy per
- app server, reused forever
- widen the range:
- net.ipv4.ip_local_port_range
- = 1024 65535
- tcp_tw_reuse = 1 for outgoing
- several IPs or ports per server
- alert on TIME_WAIT count and
- connect errors
Health and draining
Health handling matters more at this scale. Proxies check each server actively, every couple of seconds, and also passively: five errors in a row and the server is ejected for thirty seconds. On a deploy, a server drains: it stops getting new requests and finishes the ones in flight. A new server gets slow start, ramping up over a minute while its JIT compiler and caches warm. And never let health checks eject more than about a third of the fleet at once.
- active check: GET /health every
- 2 s, 3 failures → out
- passive: 5 errors in a row →
- eject for 30 s
- draining on deploy: stop new
- requests, finish in-flight
- ones for up to 30 s
- slow start: ramp a new server
- over 60 s (cold JIT, caches)
- never eject more than ~30%
Reject early
At a million requests a second, some of that traffic is bots, scrapers and floods. Reject it at the cheapest layer that can recognise it. The CDN edge handles per-IP rules and bot checks. The layer-four tier uses SYN cookies against connection floods. The proxy enforces per-key rate limits and answers four twenty-nine with a retry-after header. The deeper a bad request travels, the more it costs to refuse it.
- CDN edge: per IP rules, bot
- checks, cheapest
- L4: SYN cookies against SYN
- floods
- L7: per API key rate limits,
- 429 with Retry-After
- app: business rules only
- the deeper a bad request gets,
- the more it costs to refuse
More than one region
One region is a single point of failure, so run three, all active. Normally each serves about three hundred and thirty thousand requests a second. Size each for five hundred thousand, so if one region fails, the other two carry the full million. That means paying for one and a half times the capacity you use on a normal day, which is the price of surviving a region outage. The price data must be replicated to every region, which part four covers.
3 regions, all active normal: ~333,000 req/s each size each for 500,000 lose one region: 2 × 500,000 = 1,000,000 ✓ cost: 1.5x the capacity state: prices replicated to every region (part 4)
Say it like this
In the interview: GeoDNS picks one of three active regions. An anycast CDN finishes TLS near the user and reuses connections to us. Layer-four balancers spread connections over about fifteen layer-seven proxies. The proxies balance each request with power of two choices, over pooled keep-alive connections, so we never run out of ports. Connections get a maximum age so new servers receive traffic. Then mention health checks and draining.
- "GeoDNS picks one of 3 regions.
- Anycast CDN ends TLS near the
- user. L4 spreads connections
- over ~15 L7 proxies, which
- balance each request with
- power of two choices over
- pooled keep-alive connections.
- Connections get a max age."