Start practising

SYSTEM DESIGN · CONCEPTS · SERVICES AND HOW THEY TALK · EPISODE 10

Latency, Throughput and the Numbers to Know

Latency vs throughput, why averages lie and p99 matters, the rough cost of memory, SSD, network and cross-continent calls, how latency adds up across serial calls, and how to use these numbers in a system design interview.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
1 / 10

Press play to hear the narration, or step through at your own pace. Every picture here is the same one used in the YouTube video.

Notes

Two words

Two words first. Latency is how long one request takes, say a hundred and twenty milliseconds. Throughput is how many requests you handle per second. Think of a highway: the speed limit is latency, the number of lanes is throughput. Adding servers adds lanes. It doesn't make one car faster.

Latency vs throughput
  • latency: time for ONE request
  • e.g. 120 ms
  • throughput: requests handled
  • per second, e.g. 5,000 rps
  • a highway:
  • speed limit = latency
  • number of lanes = throughput
  • add servers → more throughput,
  • not lower latency

Percentiles

Never trust the average. If ninety-five requests take fifty milliseconds and five take two seconds, the average looks fine, but one in twenty users waits two seconds. Use percentiles: p ninety-nine is the time ninety-nine percent of requests beat. And when one page makes thirty backend calls, most page loads hit at least one slow call.

Averages lie
  • 100 requests:
  • 95 take 50 ms
  • 5 take 2,000 ms
  • average = 147 ms (looks fine)
  • p95 = 50 ms
  • p99 = 2,000 ms (the truth)
  • one page = 30 backend calls
  • → most page loads hit a slow one
  • set targets on p99

The numbers

Here are the rough numbers. Memory: nanoseconds. Reading from an SSD: about a tenth of a millisecond. A Redis call in the same data centre: under a millisecond. A simple indexed database query: a few milliseconds. A spinning disk seek: ten milliseconds. And a round trip from India to the US: about two hundred milliseconds, limited by the speed of light.

Numbers to know (rough)
L1 cache / memory   1–100 ns
read 1 MB from RAM  ~0.01 ms
SSD random read     ~0.1 ms
Redis call (same DC) ~0.5 ms
DB indexed query    1–5 ms
read 1 MB from SSD  ~1 ms
HDD seek           ~10 ms
same-region hop     ~0.5 ms
India ↔ US round trip ~200 ms

Ratios

You don't need exact values. You need the ratios. Memory is about a hundred times faster than SSD. A cache hit is roughly ten times faster than a database query. A cross-continent call costs hundreds of times a same-region call. That's why we cache hot data and keep data close to users.

Ratios that drive design
  • memory ≈ 100× faster than SSD
  • cache hit ≈ 10× faster than DB
  • cross-continent ≈ 300× a
  • same-region call
  • so:
  • cache hot data in memory
  • avoid serial cross-region calls
  • put data near users (CDN,
  • regional replicas)

It adds up

Latency adds up. Three forty-millisecond calls made one after another cost a hundred and twenty. Made in parallel, they cost forty. Watch for the N plus one problem: one query for fifty posts, then one more query per post. Batch it into a single query.

Serial vs parallel
  • 3 calls of 40 ms, one after
  • another = 120 ms
  • same 3 in parallel = 40 ms
  • N+1 query: 1 query for posts,
  • then 1 per post → 51 queries
  • fix: one batched query
  • every hop costs: fewer, closer,
  • parallel

Throughput

Throughput comes from latency too. A server with a hundred worker threads and fifty-millisecond requests handles about two thousand requests a second. Need twenty thousand at peak? About ten servers, plus headroom. Little's law ties it together: requests in flight equals arrival rate times latency.

Throughput math
  • 1 server, 50 ms per request,
  • 100 worker threads
  • → ~2,000 requests / second
  • need 20,000 rps at peak
  • → ~10 servers + headroom
  • Little's law:
  • in-flight = rate × latency
  • 20,000 × 0.05 s = 1,000 at once

Budgets

Teams turn a target into a budget. If the page must load in three hundred milliseconds, split it: network to the user, the gateway, service logic, cache and database calls, rendering, and some slack. Each hop knows how much time it may spend, and any new serial call must fit.

Budgets
  • page must load in < 300 ms
  • network to user ~80 ms
  • API gateway ~5 ms
  • service logic ~20 ms
  • cache / DB calls ~40 ms
  • render ~50 ms
  • slack ~100 ms
  • a latency budget tells each
  • team how much it can spend

Say it like this

In an interview: state a target like p ninety-nine under two hundred milliseconds. Count the serial hops in your design and price them with rough numbers: memory a fraction of a millisecond, the database a few, another region a hundred and fifty. Then fix the expensive ones: cache, parallelise, move data closer.

In the interview
  • "Is this fast enough?"
  • name the target: p99 < 200 ms
  • count the serial hops
  • memory 0.1 ms, DB 5 ms,
  • cross-region 150 ms
  • cache hot reads
  • parallelise independent calls
  • keep users near their data

Interview questions

What is the difference between latency and throughput?

Latency is the time to complete one request; throughput is how many requests are completed per unit of time. Adding servers raises throughput but does not reduce the latency of a single request.

Why use p99 latency instead of the average?

Averages hide slow outliers. p99 shows the experience of the slowest 1% of requests, and because a page often makes many backend calls, tail latency affects most users.

Roughly how long does a memory read, SSD read and cross-continent round trip take?

Memory: nanoseconds (under a microsecond). SSD random read: about 0.1 ms. A round trip between continents: roughly 100–250 ms, bounded by the speed of light in fibre.

What is the N+1 query problem?

Fetching a list with one query and then issuing one more query per item, so 50 items cause 51 round trips. Fix it with joins, batched IN queries or a data loader.

What is Little's law and why is it useful?

Items in the system = arrival rate × time each spends in it. It converts throughput and latency into concurrency, for example how many threads or connections you need.

How do you reduce latency in a design?

Cache hot data in memory, avoid serial calls by parallelising independent work, batch queries, keep data near users with CDNs and regional replicas, and remove unnecessary hops.

What is a latency budget?

A split of the end-to-end latency target across each hop (network, gateway, services, storage, rendering), so every component knows how much time it may spend.