Start practising

SYSTEM DESIGN · CONCEPTS · HOW THE WEB WORKS · EPISODE 5

Load Balancers Explained: How One Address Serves Millions

Why one server is not enough, what a load balancer does, the main algorithms, health checks, L4 vs L7, sticky sessions, and how to avoid the load balancer becoming the weak point.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
1 / 20

Press play to hear the narration, or step through at your own pace. Every picture here is the same one used in the YouTube video.

Notes

The problem

One server has two problems. A ceiling: at some point the CPU, memory, or network maxes out, and responses slow down or time out. And a single point of failure: if it crashes, or you deploy new code, everyone sees an error.

As we saw last episode, the fix is to scale out: run several copies of a stateless server. But now users need one address, and something has to decide which copy handles each request.

The idea

That's a load balancer. It sits in front, with one public address. Every request comes to it, and it forwards each one to one of the servers behind it, then passes the response back.

Its jobs: spread the load so no server is overwhelmed.

Skip servers that are broken.

And let you add, remove, or update servers without users noticing. Next, how does it choose?

Algorithm 1

Algorithm one: round robin. Take turns: A, then B, then C, then back to A. Simple and fair, when every request costs about the same and every server is the same size.

Round robin
  • A → B → C → A → B → C …
  • simple and fair
  • when requests and servers
  • are all alike

Algorithm 2

Algorithm two: weighted round robin. If server A is three times bigger, give it a weight of three, and it gets three requests for every one the others get.

Weighted round robin
  • A A A → B → C → A A A …
  • for servers of different sizes

Algorithm 3

Algorithm three: least connections. Send the next request to whichever server is least busy right now. That's better when requests vary a lot, some taking five milliseconds and some five seconds.

Least connections
  • send to the server with
  • the fewest open connections
  • good when requests vary:
  • some take 5 ms, some 5 s

Algorithm 4

Algorithm four: hashing. Hash something about the request, like the user id or IP address, and the same user always lands on the same server. With consistent hashing, adding a server only moves a small share of users. It's the base of most caches and sharded databases.

IP hash / consistent hashing
server = hash(user_id) % N

same user → same server
consistent hashing: adding a server
only moves a small share of keys

Health checks

How does it know a server is broken? Health checks. Every few seconds it calls something like /health on each server. If a server fails several checks in a row, it's taken out of rotation. When it recovers, it's added back.

L4 vs L7

Load balancers come in two kinds. Layer four balancers work with TCP: they see addresses and ports, and they're extremely fast. Layer seven balancers understand HTTP: they can read the URL, headers and cookies, so they can send /api to one group of servers and /images to another.

Layer 4 vs Layer 7
  • L4 (TCP/UDP): sees IPs and ports
  • very fast, can’t read the request
  • L7 (HTTP): reads URL, headers, cookies
  • /api → API servers
  • /images → image servers

Sticky sessions

Sometimes a server keeps session data in memory, so a user must keep returning to it. Load balancers can do that, with sticky sessions. It works, but it fights the balancing, and if that server dies the session is lost. The better fix is stateless servers, with sessions in a shared store.

Sticky sessions
  • send a user back to the same server
  • (cookie or IP based)
  • needed only for stateful servers
  • better: keep sessions in Redis

Who balances the balancer?

A sharp interviewer will ask: isn't the load balancer itself a single point of failure? Good question.

The answer: run two. An active one and a standby that watches it with heartbeats, and takes over its address if it fails. Cloud load balancers do this for you, spread across zones. And DNS can point to balancers in several regions.

In practice

In practice you'll meet software balancers like Nginx, HAProxy and Envoy, cloud ones like Amazon's Application and Network Load Balancers, and in Kubernetes, services and ingress controllers. Same ideas everywhere.

Real examples
  • hardware: F5
  • software: Nginx, HAProxy, Envoy
  • cloud: AWS ELB / ALB / NLB,
  • Google Cloud Load Balancing
  • Kubernetes: Services, Ingress

Interview questions

What is a load balancer and why use one?

A component that receives client traffic on one address and distributes it across multiple backend servers. It increases capacity, removes single points of failure (with health checks), and allows rolling deploys and autoscaling.

Compare load-balancing algorithms.

Round robin: simple rotation, good for uniform requests. Weighted round robin: proportional to server capacity. Least connections: favours the least busy server, good for uneven request times. Hash-based (IP or key, consistent hashing): keeps the same client or key on the same server, useful for caches and affinity.

L4 vs L7 load balancing?

L4 balances TCP/UDP connections using IPs and ports: very fast, protocol-agnostic, but cannot route by content. L7 terminates HTTP and can route by path, host, headers or cookies, do TLS termination, retries and rewrites, at more CPU cost.

How do you avoid the load balancer becoming a single point of failure?

Run redundant balancers (active-passive with a floating IP and heartbeats, or active-active), use managed cloud balancers spread across availability zones, and use DNS to spread traffic across regions.

What are sticky sessions and their downsides?

Routing a client to the same backend each time (via cookie or IP). Needed for in-memory sessions, but it unbalances load, and the session is lost if that server dies. Prefer stateless servers with a shared session store.