Notes
What 1M rps means
Start with the size of it. A million requests a second is one new request every microsecond. Sixty million a minute. If it ran all day, eighty-six billion requests. Real traffic has a shape, so in our example the average is three hundred thousand a second and the peak is a million. You design the system for the peak, and you try to pay for something closer to the average.
1,000,000 req/s = 1 request every microsecond = 60 million a minute = 86.4 billion a day if sustained real traffic has a shape: average 300,000 req/s peak 1,000,000 req/s design for the peak, pay for the average
The running example
We need a concrete service, so here is ours. A price API for a large online shop during a sale. Every product tile on every screen asks it for the price, the discount, stock and a delivery estimate, about two kilobytes of JSON. That is how one service ends up at a million reads a second. Prices change about two thousand times a second, and every request also records a view event for analytics.
Here is the shape we will end up with. Users reach an edge layer of CDN and load balancers. Load balancers spread requests over a fleet of stateless app servers. App servers answer from caches, and the database sits behind the caches, almost never touched on the hot path. View events go to a log like Kafka instead of the database. Each part of this series zooms into one of these boxes.
GET /v1/price/{sku}?region=IN
200 OK, about 2 KB of JSON:
price, currency, discount,
in_stock, delivery_estimate
called by every product tile on
every screen of the shop app
reads: ~1,000,000/s at peak
price updates: ~2,000/s
view events: 1 per request The core budget
The most useful number in this whole series is CPU time per request. One core gives you exactly one second of work every second. If a request costs one millisecond of CPU, a core serves a thousand requests a second. At two hundred microseconds, five thousand. At fifty, twenty thousand. Waiting for the network does not count, as long as the core can work on other requests while one waits. Part three is about making sure it can.
Now size the fleet. Cores needed equals requests per second, times CPU per request, divided by how busy you allow each core to be. Say our request costs two hundred microseconds. That is two hundred cores fully busy. We target fifty percent at peak, leaving room for spikes, deploys, garbage collection and losing a data center zone. So four hundred cores: twenty-five servers of sixteen cores, forty thousand requests a second each.
one core = 1 s of CPU per second CPU per request req/s per core 1 ms 1,000 200 µs 5,000 50 µs 20,000 10 µs 100,000 waiting on the network is not CPU: a core can serve others while one request waits
cores = rps × CPU per req ÷ target 1,000,000 × 200 µs = 200 busy cores target 50% busy at peak (room for spikes, deploys, a lost zone, GC) → 400 cores → 25 servers × 16 cores → 40,000 req/s per server same math at 1 ms: 2,000 cores
1 µs = 1 core
Here is the rule to remember. At a million requests a second, one extra microsecond per request is one more core, running all the time. One extra millisecond is a thousand more cores. Assume a core-hour costs about four cents. Then one core is about twenty-nine dollars a month, and a careless millisecond is about twenty-nine thousand dollars a month. A small inefficiency can cost more than the engineer who wrote it.
1 µs extra × 1,000,000 req/s = 1 s of CPU every second = 1 more core at peak 1 ms extra = 1,000 more cores assume a core-hour costs $0.04 1 core-month ≈ $29 1,000 cores ≈ $29,000 a month a 1 ms mistake can cost more than an engineer’s salary
Where the time goes
Where does the time go? Here is an illustrative breakdown. The kernel moving packets and system calls, TLS, parsing HTTP headers, checking an auth token, our actual business logic, encoding two kilobytes of JSON, compressing it, then logging and metrics. Notice that the business logic is one of the smallest slices. Your numbers will differ, so measure them with a profiler and a flame graph, but the lesson holds: the plumbing costs more than the feature.
kernel: packets, syscalls 40 µs
TLS decrypt + encrypt 20 µs
HTTP parsing, headers 15 µs
auth token check 25 µs
business logic 30 µs
JSON encode 2 KB 30 µs
gzip 25 µs
logging + metrics 15 µs
------
total 200 µs
measure yours: a flame graph Big-O is money
Big-O stops being an interview topic and becomes servers. Say finding the discount for a product scans a list of five thousand rules on every request. At about twenty nanoseconds per rule, that is a hundred microseconds, a hundred cores, around three thousand dollars a month. Index the rules by category in a hash map and check the ten or so that can apply, and it costs a third of a microsecond. Same answer, three hundred times cheaper.
Quadratic code is worse, and it hides well. De-duplicating a thousand items with a nested loop is about half a million comparisons, roughly a millisecond, so a thousand cores. A hash set does the same job in about twenty microseconds. In a unit test with ten items, both versions look instant. Only real data sizes, multiplied by a million requests a second, show the difference. So always ask: how big can this input get?
find the discount for a SKU: scan 5,000 rules per request 5,000 × ~20 ns = 100 µs = 100 cores ≈ $2,900/month index rules by category in a HashMap, check ~10 candidates ~0.3 µs = a third of a core same answer, 300x cheaper
de-duplicate 1,000 items with a nested loop: ~500,000 compares ≈ 1 ms per request = 1,000 cores with a HashSet: 1,000 inserts ≈ 20 µs = 20 cores fine in a unit test with 10 items, ruinous with 1,000
Allocation costs
Memory has a cost too, and in Java it shows up as garbage collection. Suppose each request allocates twenty kilobytes: copying the body, building strings, boxing numbers, creating maps and response objects. At forty thousand requests a second per server, that is eight hundred megabytes of garbage every second. A two gigabyte young generation fills every two and a half seconds. Every collection costs CPU and adds pauses to your slowest requests.
20 KB allocated per request × 40,000 req/s per server = 800 MB of garbage per second 2 GB young generation fills every ~2.5 s → a GC each time sources: copying the body, string building, boxing, big DTOs, per-request maps and lists fix: reuse buffers, encode once
Little’s Law
Next, Little’s Law: the number of requests in flight equals the arrival rate times the time each one spends in the system. A million a second at twenty milliseconds each is twenty thousand requests in flight, about eight hundred per server. Now let one dependency slow down to half a second. Same traffic, but twenty thousand in flight per server. Twenty-five times the memory, sockets and threads. Slowness, not traffic, is what usually kills a fleet.
L = λ × W in flight = rate × latency normal: 1,000,000 × 20 ms = 20,000 in flight = 800 per server a dependency slows to 500 ms: = 20,000 per server 25x the memory, sockets and threads, with no extra traffic
Rare is constant
At this scale, rare stops being rare. A one-in-a-million event happens every second. An error rate of one in ten thousand is a hundred failed requests every second. If your ninety-ninth percentile latency is fifty milliseconds, ten thousand requests every second are slower than that. That race condition you could never reproduce on your laptop? In production it will fire several times a day. Design as if every unlikely path will run.
Fan-out makes it worse. If a page calls ten backends in parallel and each is slow one percent of the time, the chance that at least one is slow is one minus point nine nine to the tenth, nearly ten percent. With a hundred backends it is sixty-three percent. Your ninety-ninth percentile becomes the user’s normal. That is why at this scale we talk about tail latency, not averages, and part five shows how to measure it properly.
1 in a million → once a second 0.01% errors → 100 per second p99 slower than X → 10,000 requests/s are slower p99.9 → 1,000 requests/s a race that needs 2 events within 1 µs: in tests never, in production several per day
one page calls 10 backends, each slow 1% of the time (p99) P(at least one slow call) = 1 − 0.99^10 ≈ 9.6% 100 backends: 1 − 0.99^100 ≈ 63% your p99 becomes the user’s typical experience
Bandwidth
Now bandwidth. A two kilobyte response a million times a second is two gigabytes a second, sixteen gigabits. Requests are not free either: seven hundred bytes of headers and cookies each is over five gigabits coming in. Compressing the JSON to about six hundred bytes cuts the outbound traffic to under five gigabits. Per server it is about six hundred and forty megabits, easy for a modern network card. The fleet total, as part six shows, is a large part of the bill.
response 2 KB × 1,000,000/s = 2 GB/s = 16 Gbit/s out request: ~700 B of headers and cookies = 5.6 Gbit/s in gzip the JSON to ~600 B → 4.8 Gbit/s out per server (40k req/s): 80 MB/s ≈ 640 Mbit/s raw fine for a 10–25 Gbit NIC
Logs and metrics
Logging deserves its own warning. A four hundred byte access log line per request is four hundred megabytes a second, about thirty-five terabytes a day at peak, ten at the average. Shipping, indexing and storing that can cost more than the servers. So you keep every error and every slow request, and sample perhaps one percent of the rest. Metrics are counted and bucketed inside the process and exported every few seconds, never sent once per request.
access log line ≈ 400 B × 1,000,000/s = 400 MB/s ≈ 34.6 TB per day at peak ≈ 10 TB/day at the average so: keep every error and slow request, sample 1% of the rest metrics: counters + histograms aggregated in process, exported every 10 s
What breaks first
When teams first push toward this scale, the app servers are rarely what breaks. They are stateless, so you add more. What breaks is everything they share. The database runs out of connections or CPU. One hot cache key melts one shard. Load balancers run out of TLS capacity. Kernel limits on files, ports and queues bite. The logging pipeline falls behind. Retries turn a small blip into an outage. And the bill arrives. Each part of the series handles some of these.
- 1 the database: connections,
- CPU, replication lag
- 2 one hot cache key or shard
- 3 load balancer and TLS CPU
- 4 kernel limits: file handles,
- ports, backlog queues
- 5 the logging pipeline
- 6 retries turning a blip into
- an outage
- 7 the bill
Rules for scale-out
From this math come a few rules. App servers hold no state that matters, so any server can answer any request and you scale by adding boxes. Per-request work is constant or logarithmic, never linear in the size of your data. Reads come from memory first. Writes are batched rather than one database write per request. Anything slow or optional moves off the hot path into a queue. And every limit you depend on has a number and an alert.
- app servers hold no state that
- matters: any server, any req
- per-request work is O(1) or
- O(log n), never O(n) of data
- reads come from memory first
- writes are batched, not 1:1
- nothing slow on the hot path:
- push it to a queue
- every limit has a number and
- an alert
Latency budget
Alongside the CPU budget, set a latency budget. Say the ninety-ninth percentile inside our data center must stay under twenty-five milliseconds. Give each layer a slice: a couple of milliseconds for the load balancers, a few for queueing in the app, one for CPU, two for the cache round trip, some for garbage collection and scheduling noise, and keep about half as headroom. The user also pays for their own network, which we cannot control, so ours must be small.
target: p99 under 25 ms inside our data center L4 + L7 load balancers 2 ms queueing in the app 3 ms app CPU 1 ms cache round trip 2 ms GC and scheduling noise 5 ms headroom 12 ms the user also pays the network
Say it like this
In an interview, open with the math. At a million requests a second, one microsecond per request is one core. I budget two hundred microseconds of CPU, so about four hundred cores at fifty percent, twenty-five servers of sixteen. Little’s Law gives twenty thousand requests in flight. Responses are two kilobytes, so I compress, cache at the edge, sample logs, and keep the database off the hot path. Then draw the design.
- "At 1M req/s, 1 µs per request
- is 1 core. I budget 200 µs of
- CPU, so ~400 cores at 50%:
- 25 servers of 16. Little’s Law
- gives 20k in flight. Responses
- are 2 KB, so I compress, cache
- at the edge, sample logs, and
- keep the DB off the hot path."