SYSTEM DESIGN · DEEP DIVE · 1M REQUESTS PER SECOND · PART 5 OF 6

Handling 1 Million Requests per Second, Part 5: Load Testing and Measuring Latency Properly

How to prove a service can take a million requests a second: generating the load from a fleet of machines, open versus closed load models, coordinated omission and how it hides slow requests, percentiles and why you never average them, realistic traffic, test shapes, finding the knee, hunting bottlenecks, testing the test, and production observability, SLOs and performance checks in CI.

Part 5 of 6 in How to Handle 1 Million Requests per Second, a deep dive in 6 videos.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
1 / 21

Press play to hear the narration, or step through at your own pace.

Notes

State the goal

Start by writing down what passing means. Not just a million requests a second, but a million sustained for an hour, with the ninety-ninth percentile under twenty-five milliseconds and the ninety-nine point ninth under a hundred, measured from the client side. Errors under one in ten thousand. Realistic keys and payloads. And the system must stay within those numbers while we kill a server, fail over a Redis shard, and deploy.

The goal, written down
  • 1,000,000 req/s sustained
  • for 1 hour
  • p99 under 25 ms, p99.9 under
  • 100 ms, measured by clients
  • errors under 0.01%
  • with realistic keys and sizes
  • and survive: one app server
  • killed, one Redis failover,
  • a rolling deploy

Generating 1M rps

No single machine can generate a million requests a second, because the client does nearly as much work as the server: TLS, HTTP and timing every response. Assume an eight-core generator can push fifty thousand a second while staying below sixty percent CPU. Then you need twenty, so run twenty-five, each sending forty thousand. Place them in the same region as the target, on a separate network path from real users, coordinated by one controller.

Some hygiene for the generators. Use tools that can send at a fixed arrival rate, such as wrk two, k6 with its arrival-rate executor, Gatling or Vegeta. Keep each generator under about sixty percent CPU, or its own delays pollute the results. Raise its file and port limits, because generators run out of ephemeral ports just like proxies. Merge latency histograms at the controller. And record the build, configuration and data set of every run.

How many generators
a generator does TLS, HTTP and
  timing for every request
assume ~50,000 req/s per
  8-core generator at 60% CPU

1,000,000 ÷ 50,000 = 20
+ 25% spare → 25 machines
  each sending 40,000 req/s

same region, separate network
path from production users
Generator hygiene
  • tools with a fixed arrival
  • rate: wrk2, k6 (arrival-rate
  • executor), Gatling, Vegeta
  • keep generator CPU under ~60%
  • raise their file and port
  • limits too (part 2, part 3)
  • sync clocks; merge histograms
  • at the controller
  • record the build, config and
  • data set of every run

Open vs closed

There are two ways to make load. In a closed model, a fixed number of virtual users each send a request, wait for the reply, then send the next. When the server slows down, they automatically send less, so the test goes easy on the server exactly when it is struggling. In an open model, requests arrive at a fixed rate no matter what, just as millions of independent users do. For a public API, test with an open model.

Two ways to make load
  • closed: N virtual users, each
  • sends again only after its
  • reply arrives
  • → server slows, load drops,
  • the test goes easy on it
  • open: requests arrive at a
  • fixed rate whatever happens
  • → like real traffic
  • for a public API: open model

Coordinated omission

This leads to coordinated omission, the most common way load tests lie. A generator sends fifty thousand a second for sixty seconds: three million requests. The server stalls for two seconds. A closed-loop tool with a hundred connections sends a hundred requests, waits, and records a hundred slow samples. That is under a thousandth of a percent, so the report says the ninety-ninth percentile was three milliseconds. In reality a hundred thousand requests were due during the stall.

Those hundred thousand requests should have been sent evenly across the two seconds, and all were answered when the stall ended. So their real latencies run from two seconds down to zero. The slowest one percent of all three million, thirty thousand requests, waited at least about one point four seconds. The fix is to measure from when each request should have been sent, which constant-rate tools do. A report with a tiny p99 and a huge maximum is a red flag.

One 2-second stall
generator: 50,000 req/s, 60 s
  = 3,000,000 requests

closed tool, 100 connections:
  100 slow samples = 0.003%
  reported p99: 3 ms ✓

reality: 100,000 requests
  were due during the stall
  = 3.3%, delayed 0–2 s
  true p99 ≈ 1.4 s ✗
Measure from the intended time
latency = reply time
        − time it SHOULD have
          been sent

requests due at t=0.0 … 2.0 s
  all answered at t=2.0 s
  → 2.0 s, 1.9 s, … 0.0 s
the slowest 1% (30,000)
  wait at least ~1.4 s

red flag: p99 3 ms, max 2 s

Percentiles

Now, what to report. The mean is a poor summary: here it is six milliseconds, while half the requests take three and one in a hundred takes forty. Report percentiles: the median, the ninety-ninth, the ninety-nine point ninth and the maximum. At a million requests a second, the ninety-nine point ninth percentile is a thousand requests every second, and a page making thirty calls sends most users through a ninety-ninth percentile on every visit.

A trap with percentiles: you cannot average them. If twenty-five servers each report their own ninety-ninth percentile, the average of those numbers is not the fleet’s. One sick server gets diluted by twenty-four healthy ones. Instead, collect histograms, add the bucket counts across servers, and then compute the percentile. HdrHistogram and Prometheus histograms work this way. Pre-computed summaries cannot be merged. And put a bucket boundary exactly at your target, twenty-five milliseconds.

One run, five numbers
mean     6 ms   (describes no one)
p50      3 ms   half are faster
p99     40 ms   1 in 100 slower
p99.9  180 ms   1 in 1,000
max    2.1 s    one unlucky user

at 1M req/s, p99.9 = 1,000
  requests every second
a page with 30 calls: most
  users hit a p99 every visit
Never average percentiles
  • 25 servers each report a p99
  • mean of the 25 p99s is not
  • the fleet p99: one bad
  • server gets diluted
  • merge histograms instead:
  • HdrHistogram, or Prometheus
  • histogram buckets summed,
  • then the percentile
  • pre-computed summaries cannot
  • be merged; put a bucket
  • edge at the SLO (25 ms)

Realistic traffic

A load test is only as honest as its traffic. Key distribution matters most, because it decides cache hit rates. Real shoppers are skewed: a few products get most of the views. Uniform random keys make caches look terrible, and hammering one key makes them look perfect. Replay anonymised production logs if you can. Use real header and payload sizes, keep TLS on, open new connections at the real rate, and include the write side: price changes and view events.

Make it look like production
  • keys: skewed like real traffic
  • (a few products get most
  • views), not uniform, not 1
  • key on repeat
  • replay anonymised real logs
  • real payload and header sizes
  • TLS on; new connections at
  • the real rate
  • regions and mobile clients
  • include the write side

Warm and cold

Test two states. Warm is the steady state with full caches, and it gives the flattering number. Cold is what happens after a deploy or a cache failover: local caches empty, one Redis shard replaced, misses rushing toward the database. Run the full rate through a deploy and a shard failover, and watch the database: does the concurrency cap hold, do misses queue politely? The capacity you can claim is the cold number, not the warm one.

Test both states
  • warm: caches full, steady
  • state, the easy number
  • cold: start with empty local
  • caches, fail over a Redis
  • shard, deploy mid-test
  • watch the DB during cold
  • phases: concurrency cap
  • holding? misses queued?
  • the cold number is the real
  • capacity

Test shapes

Use several test shapes. A ramp adds ten percent every few minutes and shows where latency starts to bend. A soak holds the target for eight to twenty-four hours to find slow leaks: memory, file handles, connections, disk filling with logs. A spike jumps from twenty to a hundred percent in thirty seconds, testing autoscaling and caches. A stress test goes past the target until something breaks, because you want to know how it fails: gracefully with fast rejections, or by collapsing.

Test shapes
  • ramp: +10% every 5 min up to
  • target; find where it bends
  • soak: target for 8–24 h; finds
  • leaks of memory, files,
  • connections, disk
  • spike: 20% → 100% in 30 s;
  • do autoscaling and caches
  • keep up?
  • stress: past target until it
  • breaks: how does it fail?

Finding the knee

Plot latency against throughput and you get a hockey stick: flat, then suddenly vertical. A simple queueing model explains why. The time spent waiting grows like utilisation divided by one minus utilisation. At fifty percent busy, a request waits about one service time. At eighty, four. At ninety, nine. At ninety-five, nineteen. That is the knee, and it is why part one targeted fifty percent. Run in production well below the knee you measured.

Why latency has a knee
simple queueing model:
  wait ≈ service × ρ ÷ (1 − ρ)

utilisation ρ   queue wait
  50%            1 × service
  80%            4 ×
  90%            9 ×
  95%           19 ×

run at ~60% of the knee:
  the headroom from part 1

Watch everything

During a run, watch everything with two checklists. For every resource, USE: utilisation, saturation and errors. That means CPU and run queue length, garbage collection time, memory, file descriptors, ports, the conntrack table, network drops, Redis CPU and database connections. For every service, RED: request rate, errors and duration as histograms. Add listen queue overflows, retry counts and the hit rate of every cache layer. The bottleneck is whatever saturates first.

What to watch during a run
  • USE per resource: utilisation,
  • saturation, errors
  • CPU and run queue, GC time,
  • memory, file descriptors,
  • ports, conntrack, NIC drops,
  • Redis CPU, DB connections
  • RED per service: rate, errors,
  • duration (histograms)
  • plus: listen overflows, retries,
  • cache hit rate per layer

Bottleneck hunt

A real campaign is a loop: run, find the first wall, fix it, run again. Here is a typical sequence. Run one bends at six hundred thousand: one Redis shard is at full CPU because of a hot key, so we add local caching. Run two bends at eight hundred and fifty thousand: proxies are dropping connection attempts, so we raise the backlog. Run three shows spikes every thirty seconds: garbage collection, so we cut allocation. Run four holds one point one five million.

A typical hunt
  • run 1: bends at 600k req/s
  • Redis shard 4 at 100% CPU:
  • a hot key → local cache
  • run 2: bends at 850k
  • proxies drop SYNs: listen
  • overflows → raise backlog
  • run 3: p99.9 spikes every 30 s
  • GC on app servers → cut
  • allocation, switch collector
  • run 4: 1.15M req/s, p99 18 ms

Test the test

Always test the test. Double the number of generators at the same total rate. If the results change, the generators were the bottleneck, not the system. Check their CPU, ports and network drops. Follow the rules of any CDN or cloud service you send load through, and get permission where needed. And never surprise production users or the teams who own shared systems like the database. A load test that causes an outage teaches the wrong lesson.

Test the test
  • double the generators: if the
  • numbers change, the
  • generators were the limit
  • check generator CPU, ports,
  • network drops
  • do not load test a CDN or
  • cloud service without its
  • rules or permission
  • never surprise production
  • users or other teams

Production telemetry

The same thinking carries into production. Metrics are histograms per route, aggregated in the process and exported every few seconds. Traces are sampled: a tenth of a percent is still a thousand traces a second, and tail-based sampling keeps every error and every slow request. Logs keep errors plus a small sample. And never label metrics by user or product ID: ten million products means ten million time series, which will take down the metrics system.

Observability at 1M req/s
  • metrics: histograms per route,
  • aggregated in process
  • traces: sample 0.1% → still
  • 1,000 traces/s at peak
  • tail-based: keep all errors
  • and slow ones
  • logs: errors + 1% sample
  • labels: never per user or per
  • SKU (10M series)

SLOs and alerts

Turn the goal into a service level objective: ninety-nine point nine percent of requests succeed in under twenty-five milliseconds, measured monthly. At an average of three hundred thousand a second, that is about seven hundred and seventy-eight billion requests a month, so the error budget is around seven hundred and seventy-eight million bad ones. Do not page on every blip. Page when the budget burns, say, fourteen times faster than sustainable for an hour.

SLOs and alerts
SLO: 99.9% of requests succeed
  in under 25 ms, per month

average 300,000 req/s
  ≈ 777.6 billion req/month
error budget 0.1%
  ≈ 777 million bad requests

alert on burn rate: budget
  spent 14x too fast for 1 h
  → page someone

Checks in CI

Finally, do not wait for a big test to find regressions. On every build, run a ten-minute test against a single server and record requests per second per core, CPU microseconds per request, bytes allocated per request and the ninety-ninth percentile. Fail the build if any gets more than five percent worse. Remember the rule from part one: one microsecond per request is one core. A check in CI is the cheapest place to save it.

Performance checks in CI
  • each build: a 10-minute test
  • against 1 server
  • record: req/s per core, CPU µs
  • per request, bytes allocated
  • per request, p99
  • fail the build on a > 5%
  • regression
  • remember: 1 µs = 1 core
  • = ~$29 a month

Say it like this

In an interview: I load test with an open model at a fixed arrival rate from about twenty-five generators, measuring latency from the intended send time to avoid coordinated omission. I report merged ninety-ninth and ninety-nine point ninth percentiles, never averages. I test with realistic skewed keys, both warm and cold, ramp until I find the knee, and run production at around sixty percent of it, with SLO burn-rate alerts.

In the interview
  • "I test with an open model at
  • a fixed rate from ~25
  • generators, measuring from the
  • intended send time to avoid
  • coordinated omission. I report
  • merged p99 and p99.9, test warm
  • and cold, ramp to the knee, and
  • run at ~60% of it."

Interview questions

How do you generate 1 million requests per second for a load test?

Use a fleet of load generators, for example 25 machines each sending 40,000 requests per second at a fixed arrival rate, coordinated by a controller that merges their latency histograms. Keep each generator below about 60% CPU and raise its port and file limits.

What is coordinated omission?

When a closed-loop load tool waits for slow responses, it stops sending during a stall, so the requests that should have been sent are never measured. Percentiles then look far better than reality. Measure latency from the intended send time with a constant-rate tool.

What is the difference between an open and a closed load model?

In a closed model a fixed set of virtual users waits for each reply before sending again, so load drops when the server slows. In an open model requests arrive at a set rate regardless, like real independent users, which is right for public APIs.

Why should you not average p99 values across servers?

The mean of per-server percentiles is not the fleet percentile and hides one bad server. Merge the underlying histograms by summing bucket counts, then compute the percentile from the merged histogram.

Why does latency rise sharply near full utilisation?

Queueing delay grows roughly with utilisation divided by one minus utilisation, so waiting is about 1 service time at 50%, 9 at 90% and 19 at 95%. That creates a knee, so production should run well below it.

What makes load test traffic realistic?

A key distribution as skewed as production, real payload and header sizes, TLS, realistic new-connection rates, the write side, and ideally replayed anonymised production logs. Uniform keys or a single key give misleading cache hit rates.

How do you check that the load generator is not the bottleneck?

Spread the same total rate over twice as many generators and compare results, and watch generator CPU, ports and network drops. If results improve with more generators, the test rig was limiting.

How do you alert on an SLO at this scale?

Define an error budget, such as 0.1% of monthly requests failing or exceeding 25 ms, and alert on burn rate, for example when the budget is being spent 14 times faster than sustainable over an hour, rather than on single errors.