Start practising

SYSTEM DESIGN · BUILDING BLOCKS · 1 OF 14

Load Balancer: Internals, Algorithms, Failover and Sizing

The load balancer as you would draw and defend it in a design interview: what happens to one request inside an L7 proxy, L4 versus L7, the balancing algorithms and when each wins, health checks and connection draining, how the balancer itself stays up, two-tier and global load balancing, and how to size the tier.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
1 / 16

Press play to hear the narration, or step through at your own pace.

Notes

The problem

Start with the problem. One app server has a ceiling: so many cores, so much memory, so many open connections. It's also a single point of failure, and every deploy restarts it. The fix is many identical stateless servers. But clients need one address to talk to, and something has to spread the work. That something is the load balancer.

Why one server is not enough
  • capacity: one box tops out
  • (CPU, memory, connections)
  • availability: it dies,
  • the whole product is down
  • deploys: restarting it
  • means downtime
  • fix: many identical servers
  • and something to spread load

What it does

The load balancer does four things. It accepts traffic on one stable virtual IP. It chooses a healthy backend. It forwards the request and the response. And it protects the pool: it stops sending to dead servers, can shed load, and usually terminates TLS. Clients never learn how many servers exist, so you can add and remove them freely.

The job, in four verbs
  • accept: one virtual IP or
  • hostname for all clients
  • choose: pick a healthy backend
  • forward: proxy the request
  • and the response
  • protect: drop dead servers,
  • shed load, terminate TLS

L4 vs L7

There are two kinds. A layer four balancer sees only IP addresses and ports. It forwards whole TCP connections, which is extremely fast, millions of packets a second, but it can't read paths or cookies. A layer seven balancer terminates TLS and parses HTTP, so it can route /api to one pool and /images to another, retry, and balance each gRPC call. It costs more CPU per request.

L4 vs L7
  • L4 (TCP/UDP):
  • sees IPs and ports only
  • forwards connections
  • millions of packets/s
  • no TLS, no paths, no cookies
  • L7 (HTTP/gRPC):
  • terminates TLS, reads request
  • routes by path, header, host
  • retries, gRPC per-request LB
  • more CPU per request

Inside one request

Follow one request through an L7 balancer like Envoy or Nginx. It accepts the connection and finishes the TLS handshake. It parses the host, path and headers, matches a route to a backend pool, and picks a healthy server. It reuses a warm keep-alive connection to that server, adds headers like X-Forwarded-For and a request id, waits with a timeout, and streams the answer back.

One request inside an L7 LB
1 accept TCP, finish TLS
2 parse HTTP: host, path, headers
3 match a route → backend pool
4 pick a healthy backend (algo)
5 reuse a warm upstream connection
6 add X-Forwarded-For, request id
7 wait with a timeout, maybe retry
8 stream response back, log it

Algorithms

Now the algorithms. Round robin hands requests out in turn; weighted round robin gives bigger machines more. That's fine when every request costs about the same. When cost varies, like a mix of quick reads and slow reports, use least connections. At large scale, power of two choices samples two servers at random and picks the less busy one, which avoids everyone stampeding the same least-loaded box.

Balancing algorithms
  • round robin: next in turn
  • weighted RR: big boxes get more
  • least connections: fewest
  • in-flight requests
  • power of two choices: sample 2,
  • pick the less busy one
  • hash (IP, user, key): same
  • client → same server

Affinity

Sometimes you want the same client on the same server: a WebSocket room, a local cache, an in-memory session you haven't moved out yet. Use a sticky cookie or hash a key like the user id. Plain hash mod N moves almost every key when a server is added, so use consistent hashing, from Concepts episode twenty-two. Prefer stateless servers, though; stickiness makes failover and scaling harder.

Sticky routing
  • cookie stickiness: LB sets a
  • cookie naming the backend
  • hash routing: hash(key) picks
  • the server (cache locality,
  • WebSocket rooms, shards)
  • plain mod N reshuffles all keys
  • when N changes → use
  • consistent hashing (Concepts 22)

Health checks

How does it know a server is dead? Two ways. Active checks probe a health endpoint every few seconds: three failures and the server is out, two passes and it's back. Passive checks watch real traffic: five errors in a row and the server is ejected for thirty seconds. The pool itself usually comes from service discovery, so new servers join without anyone editing a config file.

Health checking
  • active: probe /healthz every
  • 5 s; 3 failures → out,
  • 2 passes → back in
  • passive (outlier detection):
  • 5 errors in a row → eject
  • for 30 s
  • pool from a registry or
  • service discovery (Consul,
  • Kubernetes endpoints)

Draining

Deploys are where careless setups drop requests. Before stopping a server, drain it: the balancer stops sending new requests but lets in-flight ones finish, say for thirty seconds. When a new server joins, slow start ramps its weight up over a minute so its cold caches don't get flattened. And make the health endpoint check real readiness, not just that the process is running.

Deploys without errors
  • connection draining:
  • mark server "draining"
  • no new requests
  • finish in-flight (e.g. 30 s)
  • then stop it
  • slow start: ramp a new server
  • from 0 to full weight over
  • 30-60 s (cold caches, JIT)
  • health endpoint ≠ "process up"

When the LB dies

The balancer is now your single point of failure, so it comes in pairs. In active-passive, two machines exchange heartbeats, and if the active one goes quiet the standby claims the virtual IP, usually within a few seconds. In active-active, several balancers all take traffic, spread by DNS or by routers using equal-cost multipath. Managed cloud balancers do this for you, but they still have quotas and warm-up behaviour.

Keeping the LB itself up
  • active-passive pair:
  • heartbeat (VRRP/keepalived)
  • standby claims the virtual IP
  • failover in ~1-3 s
  • active-active:
  • several LBs behind DNS
  • or ECMP / anycast routing
  • cloud LBs do this for you
  • (but they still have limits)

Scaling the tier

One box can't terminate TLS for a whole large site, so big systems use two tiers. A layer four tier, spread by routers with equal-cost multipath or anycast, hashes each connection to one of many layer seven proxies. The proxies do the expensive work: TLS, routing, retries. Need more capacity? Add proxies. DNS hands out several entry IPs with a short TTL.

Scaling the balancer
  • tier 1: L4, ECMP or anycast
  • very cheap per packet
  • tier 2: fleet of L7 proxies
  • TLS, routing, retries
  • add L7 nodes horizontally;
  • L4 hashes the connection
  • (5-tuple) to keep it on one
  • DNS: several IPs, short TTL

Failure modes

Name the failure modes before the interviewer does. Cascades: one of three servers dies, the other two take fifty percent more and fall over too, so keep spare capacity. Retry storms: retrying every timeout can double or triple load, so cap retries with a budget, say ten percent. And flapping checks: if half the pool looks unhealthy at once, it's probably the check, so a panic threshold keeps routing to everyone.

Failure modes to name
  • cascade: 1 of 3 dies, others
  • take +50% and fall over
  • → keep N+1 headroom
  • retry storm: LB retries on
  • every timeout → 2-3x load
  • → retry budget (e.g. 10%)
  • flapping health checks
  • → hysteresis, panic threshold
  • (ignore checks if 50% fail)

Global

Across regions there's one more layer. Geo or latency-based DNS answers with the nearest healthy region's address, but failover is only as fast as clients respect the TTL. Anycast advertises the same IP from every location, and internet routing delivers you to the nearest one, so failover takes seconds. Inside each region, it's the same balancer story again.

Global load balancing
  • GeoDNS / latency DNS:
  • answer with nearest region
  • failover = change answer
  • (slow: TTLs, caching)
  • anycast: same IP everywhere,
  • BGP sends you to the nearest
  • edge; failover in seconds
  • then a regional LB as before

Sizing

Sizing is simple arithmetic with stated assumptions. Say sixty thousand HTTPS requests a second at peak. If one eight-core proxy handles about fifteen thousand a second with TLS, which you'd benchmark, you need four, so run six for failures. If an app server handles four hundred a second, that's a hundred and fifty, plus thirty percent headroom. And by Little's law, sixty thousand a second times two hundred milliseconds is twelve thousand requests in flight.

Sizing the tier
peak: 60,000 req/s HTTPS
one L7 proxy (8 vCPU): ~15,000
  req/s with TLS (measure it!)
→ 4 proxies, run 6 (N+2)

app server: ~400 req/s each
→ 150 servers + 30% = ~200

conns: 60k rps × 0.2 s = 12k
  in flight (Little's law)

Say it like this

In the interview, draw DNS, a redundant balancer, and a pool of stateless servers. Then justify it in a sentence each: layer seven because you need TLS termination and path routing; least connections or power of two choices because request cost varies; active plus passive health checks; a redundant pair or a managed balancer so it isn't a single point of failure. Mention draining, headroom and retry budgets, and you sound like you've run one.

In the interview
  • draw: DNS → L4/L7 LB pair
  • → stateless app pool
  • say: L7 for TLS + path routing
  • algorithm: least-conn or P2C
  • health: active + outlier
  • HA: pair with floating IP
  • or managed (ALB, NLB, GCLB)
  • mention: draining, headroom,
  • retry budget, global DNS

Interview questions

What is the difference between an L4 and an L7 load balancer?

An L4 balancer forwards TCP or UDP connections using only IPs and ports, so it is very fast but cannot see requests. An L7 balancer terminates TLS and parses HTTP, so it can route by path, host or header, retry, and balance individual gRPC calls, at a higher CPU cost.

When would you choose least connections over round robin?

When requests vary a lot in cost or duration, because round robin keeps sending work to a server stuck on slow requests while least connections favours the server with the fewest requests in flight.

What is power of two choices and why is it used?

The balancer samples two backends at random and sends the request to the less loaded one. It gets close to the best possible spread without global state, and avoids many balancers all piling onto the same least-loaded server.

How does a load balancer detect a failed server?

Active health checks probe an endpoint on a schedule and remove a server after a few consecutive failures, and passive outlier detection ejects a server that returns repeated errors on real traffic, readmitting it after a cool-off.

How do you avoid the load balancer being a single point of failure?

Run it redundantly, either as an active-passive pair sharing a floating virtual IP via heartbeats, or as an active-active fleet spread by DNS, ECMP or anycast; managed cloud balancers provide this built in.

What is connection draining?

Before removing a server, the balancer stops giving it new requests but lets in-flight requests finish for a grace period, so deploys and scale-downs do not cause errors.

Why are sticky sessions discouraged, and what do you do instead?

They tie users to one server, which skews load and loses sessions when that server dies. Keep servers stateless by moving sessions to a shared store such as Redis or a signed token, and use consistent hashing only when locality really matters.

How do you size a load balancer tier?

Take peak requests per second, divide by the measured capacity of one balancer node with TLS, and add headroom for at least one or two node failures; then check connections in flight using peak rate times average latency.

How does load balancing work across regions?

GeoDNS or latency-based DNS returns the nearest healthy region, or anycast advertises one IP from many locations so routing picks the closest; each region then has its own regional load balancers in front of its servers.