SYSTEM DESIGN · DEEP DIVE · 1M REQUESTS PER SECOND · PART 1 OF 6

Handling 1 Million Requests per Second, Part 1: The Scale and the Per-Request Budget

What a million requests a second really means: one request every microsecond, the CPU budget per request, the rule that one microsecond per request costs one core, how Big-O and memory allocation turn into servers and money, Little’s Law for requests in flight, why one-in-a-million events happen every second, and the bandwidth and logging bill before any code is written.

Reported atApple

Part 1 of 6 in How to Handle 1 Million Requests per Second, a deep dive in 6 videos.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
1 / 22

Press play to hear the narration, or step through at your own pace.

Notes

What 1M rps means

Start with the size of it. A million requests a second is one new request every microsecond. Sixty million a minute. If it ran all day, eighty-six billion requests. Real traffic has a shape, so in our example the average is three hundred thousand a second and the peak is a million. You design the system for the peak, and you try to pay for something closer to the average.

The size of a second
1,000,000 req/s
= 1 request every microsecond
= 60 million a minute
= 86.4 billion a day if sustained

real traffic has a shape:
  average  300,000 req/s
  peak   1,000,000 req/s

design for the peak,
pay for the average

The running example

We need a concrete service, so here is ours. A price API for a large online shop during a sale. Every product tile on every screen asks it for the price, the discount, stock and a delivery estimate, about two kilobytes of JSON. That is how one service ends up at a million reads a second. Prices change about two thousand times a second, and every request also records a view event for analytics.

Here is the shape we will end up with. Users reach an edge layer of CDN and load balancers. Load balancers spread requests over a fleet of stateless app servers. App servers answer from caches, and the database sits behind the caches, almost never touched on the hot path. View events go to a log like Kafka instead of the database. Each part of this series zooms into one of these boxes.

Our service: the price API
GET /v1/price/{sku}?region=IN

200 OK, about 2 KB of JSON:
  price, currency, discount,
  in_stock, delivery_estimate

called by every product tile on
  every screen of the shop app
reads: ~1,000,000/s at peak
price updates: ~2,000/s
view events: 1 per request

The core budget

The most useful number in this whole series is CPU time per request. One core gives you exactly one second of work every second. If a request costs one millisecond of CPU, a core serves a thousand requests a second. At two hundred microseconds, five thousand. At fifty, twenty thousand. Waiting for the network does not count, as long as the core can work on other requests while one waits. Part three is about making sure it can.

Now size the fleet. Cores needed equals requests per second, times CPU per request, divided by how busy you allow each core to be. Say our request costs two hundred microseconds. That is two hundred cores fully busy. We target fifty percent at peak, leaving room for spikes, deploys, garbage collection and losing a data center zone. So four hundred cores: twenty-five servers of sixteen cores, forty thousand requests a second each.

Requests per core
one core = 1 s of CPU per second

CPU per request   req/s per core
  1 ms              1,000
  200 µs            5,000
  50 µs            20,000
  10 µs           100,000

waiting on the network is not
CPU: a core can serve others
while one request waits
Fleet size
cores = rps × CPU per req ÷ target

1,000,000 × 200 µs = 200 busy cores
target 50% busy at peak
  (room for spikes, deploys,
   a lost zone, GC)
→ 400 cores
→ 25 servers × 16 cores
→ 40,000 req/s per server

same math at 1 ms: 2,000 cores

1 µs = 1 core

Here is the rule to remember. At a million requests a second, one extra microsecond per request is one more core, running all the time. One extra millisecond is a thousand more cores. Assume a core-hour costs about four cents. Then one core is about twenty-nine dollars a month, and a careless millisecond is about twenty-nine thousand dollars a month. A small inefficiency can cost more than the engineer who wrote it.

One microsecond = one core
1 µs extra × 1,000,000 req/s
  = 1 s of CPU every second
  = 1 more core at peak
1 ms extra = 1,000 more cores

assume a core-hour costs $0.04
  1 core-month ≈ $29
  1,000 cores ≈ $29,000 a month

a 1 ms mistake can cost more
than an engineer’s salary

Where the time goes

Where does the time go? Here is an illustrative breakdown. The kernel moving packets and system calls, TLS, parsing HTTP headers, checking an auth token, our actual business logic, encoding two kilobytes of JSON, compressing it, then logging and metrics. Notice that the business logic is one of the smallest slices. Your numbers will differ, so measure them with a profiler and a flame graph, but the lesson holds: the plumbing costs more than the feature.

Where 200 µs goes (example)
kernel: packets, syscalls   40 µs
TLS decrypt + encrypt       20 µs
HTTP parsing, headers       15 µs
auth token check            25 µs
business logic              30 µs
JSON encode 2 KB            30 µs
gzip                        25 µs
logging + metrics           15 µs
                           ------
total                      200 µs
measure yours: a flame graph

Big-O is money

Big-O stops being an interview topic and becomes servers. Say finding the discount for a product scans a list of five thousand rules on every request. At about twenty nanoseconds per rule, that is a hundred microseconds, a hundred cores, around three thousand dollars a month. Index the rules by category in a hash map and check the ten or so that can apply, and it costs a third of a microsecond. Same answer, three hundred times cheaper.

Quadratic code is worse, and it hides well. De-duplicating a thousand items with a nested loop is about half a million comparisons, roughly a millisecond, so a thousand cores. A hash set does the same job in about twenty microseconds. In a unit test with ten items, both versions look instant. Only real data sizes, multiplied by a million requests a second, show the difference. So always ask: how big can this input get?

Big-O becomes servers
find the discount for a SKU:

scan 5,000 rules per request
  5,000 × ~20 ns = 100 µs
  = 100 cores ≈ $2,900/month

index rules by category in a
HashMap, check ~10 candidates
  ~0.3 µs = a third of a core

same answer, 300x cheaper
Quadratic is a fleet
de-duplicate 1,000 items with
a nested loop: ~500,000 compares
  ≈ 1 ms per request
  = 1,000 cores

with a HashSet: 1,000 inserts
  ≈ 20 µs = 20 cores

fine in a unit test with 10
items, ruinous with 1,000

Allocation costs

Memory has a cost too, and in Java it shows up as garbage collection. Suppose each request allocates twenty kilobytes: copying the body, building strings, boxing numbers, creating maps and response objects. At forty thousand requests a second per server, that is eight hundred megabytes of garbage every second. A two gigabyte young generation fills every two and a half seconds. Every collection costs CPU and adds pauses to your slowest requests.

Memory churn
20 KB allocated per request
× 40,000 req/s per server
= 800 MB of garbage per second

2 GB young generation fills
  every ~2.5 s → a GC each time

sources: copying the body, string
  building, boxing, big DTOs,
  per-request maps and lists
fix: reuse buffers, encode once

Little’s Law

Next, Little’s Law: the number of requests in flight equals the arrival rate times the time each one spends in the system. A million a second at twenty milliseconds each is twenty thousand requests in flight, about eight hundred per server. Now let one dependency slow down to half a second. Same traffic, but twenty thousand in flight per server. Twenty-five times the memory, sockets and threads. Slowness, not traffic, is what usually kills a fleet.

Requests in flight
L = λ × W
in flight = rate × latency

normal: 1,000,000 × 20 ms
  = 20,000 in flight
  = 800 per server

a dependency slows to 500 ms:
  = 20,000 per server
  25x the memory, sockets and
  threads, with no extra traffic

Rare is constant

At this scale, rare stops being rare. A one-in-a-million event happens every second. An error rate of one in ten thousand is a hundred failed requests every second. If your ninety-ninth percentile latency is fifty milliseconds, ten thousand requests every second are slower than that. That race condition you could never reproduce on your laptop? In production it will fire several times a day. Design as if every unlikely path will run.

Fan-out makes it worse. If a page calls ten backends in parallel and each is slow one percent of the time, the chance that at least one is slow is one minus point nine nine to the tenth, nearly ten percent. With a hundred backends it is sixty-three percent. Your ninety-ninth percentile becomes the user’s normal. That is why at this scale we talk about tail latency, not averages, and part five shows how to measure it properly.

Rare events, every second
1 in a million → once a second
0.01% errors   → 100 per second

p99 slower than X
  → 10,000 requests/s are slower
p99.9 → 1,000 requests/s

a race that needs 2 events
within 1 µs: in tests never,
in production several per day
Fan-out makes tails common
one page calls 10 backends,
each slow 1% of the time (p99)

P(at least one slow call)
  = 1 − 0.99^10 ≈ 9.6%

100 backends: 1 − 0.99^100
  ≈ 63%

your p99 becomes the user’s
typical experience

Bandwidth

Now bandwidth. A two kilobyte response a million times a second is two gigabytes a second, sixteen gigabits. Requests are not free either: seven hundred bytes of headers and cookies each is over five gigabits coming in. Compressing the JSON to about six hundred bytes cuts the outbound traffic to under five gigabits. Per server it is about six hundred and forty megabits, easy for a modern network card. The fleet total, as part six shows, is a large part of the bill.

Bytes on the wire
response 2 KB × 1,000,000/s
  = 2 GB/s = 16 Gbit/s out
request: ~700 B of headers and
  cookies = 5.6 Gbit/s in

gzip the JSON to ~600 B
  → 4.8 Gbit/s out

per server (40k req/s):
  80 MB/s ≈ 640 Mbit/s raw
  fine for a 10–25 Gbit NIC

Logs and metrics

Logging deserves its own warning. A four hundred byte access log line per request is four hundred megabytes a second, about thirty-five terabytes a day at peak, ten at the average. Shipping, indexing and storing that can cost more than the servers. So you keep every error and every slow request, and sample perhaps one percent of the rest. Metrics are counted and bucketed inside the process and exported every few seconds, never sent once per request.

Observability at scale
access log line ≈ 400 B
× 1,000,000/s = 400 MB/s
≈ 34.6 TB per day at peak
≈ 10 TB/day at the average

so: keep every error and slow
  request, sample 1% of the rest
metrics: counters + histograms
  aggregated in process,
  exported every 10 s

What breaks first

When teams first push toward this scale, the app servers are rarely what breaks. They are stateless, so you add more. What breaks is everything they share. The database runs out of connections or CPU. One hot cache key melts one shard. Load balancers run out of TLS capacity. Kernel limits on files, ports and queues bite. The logging pipeline falls behind. Retries turn a small blip into an outage. And the bill arrives. Each part of the series handles some of these.

What breaks first
  • 1 the database: connections,
  • CPU, replication lag
  • 2 one hot cache key or shard
  • 3 load balancer and TLS CPU
  • 4 kernel limits: file handles,
  • ports, backlog queues
  • 5 the logging pipeline
  • 6 retries turning a blip into
  • an outage
  • 7 the bill

Rules for scale-out

From this math come a few rules. App servers hold no state that matters, so any server can answer any request and you scale by adding boxes. Per-request work is constant or logarithmic, never linear in the size of your data. Reads come from memory first. Writes are batched rather than one database write per request. Anything slow or optional moves off the hot path into a queue. And every limit you depend on has a number and an alert.

Rules for scaling out
  • app servers hold no state that
  • matters: any server, any req
  • per-request work is O(1) or
  • O(log n), never O(n) of data
  • reads come from memory first
  • writes are batched, not 1:1
  • nothing slow on the hot path:
  • push it to a queue
  • every limit has a number and
  • an alert

Latency budget

Alongside the CPU budget, set a latency budget. Say the ninety-ninth percentile inside our data center must stay under twenty-five milliseconds. Give each layer a slice: a couple of milliseconds for the load balancers, a few for queueing in the app, one for CPU, two for the cache round trip, some for garbage collection and scheduling noise, and keep about half as headroom. The user also pays for their own network, which we cannot control, so ours must be small.

Latency budget (server side)
target: p99 under 25 ms inside
our data center

L4 + L7 load balancers   2 ms
queueing in the app      3 ms
app CPU                  1 ms
cache round trip         2 ms
GC and scheduling noise  5 ms
headroom                12 ms

the user also pays the network

Say it like this

In an interview, open with the math. At a million requests a second, one microsecond per request is one core. I budget two hundred microseconds of CPU, so about four hundred cores at fifty percent, twenty-five servers of sixteen. Little’s Law gives twenty thousand requests in flight. Responses are two kilobytes, so I compress, cache at the edge, sample logs, and keep the database off the hot path. Then draw the design.

In the interview
  • "At 1M req/s, 1 µs per request
  • is 1 core. I budget 200 µs of
  • CPU, so ~400 cores at 50%:
  • 25 servers of 16. Little’s Law
  • gives 20k in flight. Responses
  • are 2 KB, so I compress, cache
  • at the edge, sample logs, and
  • keep the DB off the hot path."

Interview questions

How many servers do you need for 1 million requests per second?

Measure CPU time per request, then cores = requests per second × CPU seconds per request ÷ target utilisation. At 200 microseconds and 50% target that is 400 cores, for example 25 servers with 16 cores, each serving 40,000 requests a second.

Why does one microsecond per request matter at this scale?

A million requests a second times one microsecond is one second of CPU every second, which is one full core. One extra millisecond per request therefore needs about a thousand more cores, a large monthly cost.

What is Little’s Law and how do you use it here?

Requests in flight equal arrival rate times time in the system. At 1 million per second and 20 ms that is 20,000 concurrent requests. If latency rises to 500 ms, concurrency rises 25 times, which is why slow dependencies exhaust threads, sockets and memory.

Why do Big-O choices matter more at high request rates?

Per-request work is multiplied by a million every second. A linear scan of 5,000 rules costing 100 microseconds becomes 100 cores, while an indexed lookup costing under a microsecond needs less than one, for the same result.

How much bandwidth does 1 million requests per second use?

Multiply by payload size. Two kilobyte responses are 2 GB/s, or 16 Gbit/s outbound, plus inbound headers. Compression to about 600 bytes cuts that to under 5 Gbit/s, which also lowers egress cost.

Why is logging every request a problem at 1 million requests per second?

A 400 byte line per request is 400 MB/s, around 35 TB a day at peak, which is expensive to ship, index and store. Keep all errors and slow requests, sample the rest, and aggregate metrics in process.

Why does allocation per request matter in a JVM service?

Allocated bytes per request times request rate is the garbage rate. 20 KB at 40,000 requests a second is 800 MB/s, which triggers frequent young collections that cost CPU and add tail latency. Reuse buffers and avoid needless copies.

What does a one-in-a-million event mean at this scale?

It happens about once a second. Error rates, races and slow paths that never show up in testing occur continuously, so they need handling, alerts and limits rather than hope.