Notes
State the goal
Start by writing down what passing means. Not just a million requests a second, but a million sustained for an hour, with the ninety-ninth percentile under twenty-five milliseconds and the ninety-nine point ninth under a hundred, measured from the client side. Errors under one in ten thousand. Realistic keys and payloads. And the system must stay within those numbers while we kill a server, fail over a Redis shard, and deploy.
- 1,000,000 req/s sustained
- for 1 hour
- p99 under 25 ms, p99.9 under
- 100 ms, measured by clients
- errors under 0.01%
- with realistic keys and sizes
- and survive: one app server
- killed, one Redis failover,
- a rolling deploy
Generating 1M rps
No single machine can generate a million requests a second, because the client does nearly as much work as the server: TLS, HTTP and timing every response. Assume an eight-core generator can push fifty thousand a second while staying below sixty percent CPU. Then you need twenty, so run twenty-five, each sending forty thousand. Place them in the same region as the target, on a separate network path from real users, coordinated by one controller.
Some hygiene for the generators. Use tools that can send at a fixed arrival rate, such as wrk two, k6 with its arrival-rate executor, Gatling or Vegeta. Keep each generator under about sixty percent CPU, or its own delays pollute the results. Raise its file and port limits, because generators run out of ephemeral ports just like proxies. Merge latency histograms at the controller. And record the build, configuration and data set of every run.
a generator does TLS, HTTP and timing for every request assume ~50,000 req/s per 8-core generator at 60% CPU 1,000,000 ÷ 50,000 = 20 + 25% spare → 25 machines each sending 40,000 req/s same region, separate network path from production users
- tools with a fixed arrival
- rate: wrk2, k6 (arrival-rate
- executor), Gatling, Vegeta
- keep generator CPU under ~60%
- raise their file and port
- limits too (part 2, part 3)
- sync clocks; merge histograms
- at the controller
- record the build, config and
- data set of every run
Open vs closed
There are two ways to make load. In a closed model, a fixed number of virtual users each send a request, wait for the reply, then send the next. When the server slows down, they automatically send less, so the test goes easy on the server exactly when it is struggling. In an open model, requests arrive at a fixed rate no matter what, just as millions of independent users do. For a public API, test with an open model.
- closed: N virtual users, each
- sends again only after its
- reply arrives
- → server slows, load drops,
- the test goes easy on it
- open: requests arrive at a
- fixed rate whatever happens
- → like real traffic
- for a public API: open model
Coordinated omission
This leads to coordinated omission, the most common way load tests lie. A generator sends fifty thousand a second for sixty seconds: three million requests. The server stalls for two seconds. A closed-loop tool with a hundred connections sends a hundred requests, waits, and records a hundred slow samples. That is under a thousandth of a percent, so the report says the ninety-ninth percentile was three milliseconds. In reality a hundred thousand requests were due during the stall.
Those hundred thousand requests should have been sent evenly across the two seconds, and all were answered when the stall ended. So their real latencies run from two seconds down to zero. The slowest one percent of all three million, thirty thousand requests, waited at least about one point four seconds. The fix is to measure from when each request should have been sent, which constant-rate tools do. A report with a tiny p99 and a huge maximum is a red flag.
generator: 50,000 req/s, 60 s = 3,000,000 requests closed tool, 100 connections: 100 slow samples = 0.003% reported p99: 3 ms ✓ reality: 100,000 requests were due during the stall = 3.3%, delayed 0–2 s true p99 ≈ 1.4 s ✗
latency = reply time
− time it SHOULD have
been sent
requests due at t=0.0 … 2.0 s
all answered at t=2.0 s
→ 2.0 s, 1.9 s, … 0.0 s
the slowest 1% (30,000)
wait at least ~1.4 s
red flag: p99 3 ms, max 2 s Percentiles
Now, what to report. The mean is a poor summary: here it is six milliseconds, while half the requests take three and one in a hundred takes forty. Report percentiles: the median, the ninety-ninth, the ninety-nine point ninth and the maximum. At a million requests a second, the ninety-nine point ninth percentile is a thousand requests every second, and a page making thirty calls sends most users through a ninety-ninth percentile on every visit.
A trap with percentiles: you cannot average them. If twenty-five servers each report their own ninety-ninth percentile, the average of those numbers is not the fleet’s. One sick server gets diluted by twenty-four healthy ones. Instead, collect histograms, add the bucket counts across servers, and then compute the percentile. HdrHistogram and Prometheus histograms work this way. Pre-computed summaries cannot be merged. And put a bucket boundary exactly at your target, twenty-five milliseconds.
mean 6 ms (describes no one) p50 3 ms half are faster p99 40 ms 1 in 100 slower p99.9 180 ms 1 in 1,000 max 2.1 s one unlucky user at 1M req/s, p99.9 = 1,000 requests every second a page with 30 calls: most users hit a p99 every visit
- 25 servers each report a p99
- mean of the 25 p99s is not
- the fleet p99: one bad
- server gets diluted
- merge histograms instead:
- HdrHistogram, or Prometheus
- histogram buckets summed,
- then the percentile
- pre-computed summaries cannot
- be merged; put a bucket
- edge at the SLO (25 ms)
Realistic traffic
A load test is only as honest as its traffic. Key distribution matters most, because it decides cache hit rates. Real shoppers are skewed: a few products get most of the views. Uniform random keys make caches look terrible, and hammering one key makes them look perfect. Replay anonymised production logs if you can. Use real header and payload sizes, keep TLS on, open new connections at the real rate, and include the write side: price changes and view events.
- keys: skewed like real traffic
- (a few products get most
- views), not uniform, not 1
- key on repeat
- replay anonymised real logs
- real payload and header sizes
- TLS on; new connections at
- the real rate
- regions and mobile clients
- include the write side
Warm and cold
Test two states. Warm is the steady state with full caches, and it gives the flattering number. Cold is what happens after a deploy or a cache failover: local caches empty, one Redis shard replaced, misses rushing toward the database. Run the full rate through a deploy and a shard failover, and watch the database: does the concurrency cap hold, do misses queue politely? The capacity you can claim is the cold number, not the warm one.
- warm: caches full, steady
- state, the easy number
- cold: start with empty local
- caches, fail over a Redis
- shard, deploy mid-test
- watch the DB during cold
- phases: concurrency cap
- holding? misses queued?
- the cold number is the real
- capacity
Test shapes
Use several test shapes. A ramp adds ten percent every few minutes and shows where latency starts to bend. A soak holds the target for eight to twenty-four hours to find slow leaks: memory, file handles, connections, disk filling with logs. A spike jumps from twenty to a hundred percent in thirty seconds, testing autoscaling and caches. A stress test goes past the target until something breaks, because you want to know how it fails: gracefully with fast rejections, or by collapsing.
- ramp: +10% every 5 min up to
- target; find where it bends
- soak: target for 8–24 h; finds
- leaks of memory, files,
- connections, disk
- spike: 20% → 100% in 30 s;
- do autoscaling and caches
- keep up?
- stress: past target until it
- breaks: how does it fail?
Finding the knee
Plot latency against throughput and you get a hockey stick: flat, then suddenly vertical. A simple queueing model explains why. The time spent waiting grows like utilisation divided by one minus utilisation. At fifty percent busy, a request waits about one service time. At eighty, four. At ninety, nine. At ninety-five, nineteen. That is the knee, and it is why part one targeted fifty percent. Run in production well below the knee you measured.
simple queueing model: wait ≈ service × ρ ÷ (1 − ρ) utilisation ρ queue wait 50% 1 × service 80% 4 × 90% 9 × 95% 19 × run at ~60% of the knee: the headroom from part 1
Watch everything
During a run, watch everything with two checklists. For every resource, USE: utilisation, saturation and errors. That means CPU and run queue length, garbage collection time, memory, file descriptors, ports, the conntrack table, network drops, Redis CPU and database connections. For every service, RED: request rate, errors and duration as histograms. Add listen queue overflows, retry counts and the hit rate of every cache layer. The bottleneck is whatever saturates first.
- USE per resource: utilisation,
- saturation, errors
- CPU and run queue, GC time,
- memory, file descriptors,
- ports, conntrack, NIC drops,
- Redis CPU, DB connections
- RED per service: rate, errors,
- duration (histograms)
- plus: listen overflows, retries,
- cache hit rate per layer
Bottleneck hunt
A real campaign is a loop: run, find the first wall, fix it, run again. Here is a typical sequence. Run one bends at six hundred thousand: one Redis shard is at full CPU because of a hot key, so we add local caching. Run two bends at eight hundred and fifty thousand: proxies are dropping connection attempts, so we raise the backlog. Run three shows spikes every thirty seconds: garbage collection, so we cut allocation. Run four holds one point one five million.
- run 1: bends at 600k req/s
- Redis shard 4 at 100% CPU:
- a hot key → local cache
- run 2: bends at 850k
- proxies drop SYNs: listen
- overflows → raise backlog
- run 3: p99.9 spikes every 30 s
- GC on app servers → cut
- allocation, switch collector
- run 4: 1.15M req/s, p99 18 ms
Test the test
Always test the test. Double the number of generators at the same total rate. If the results change, the generators were the bottleneck, not the system. Check their CPU, ports and network drops. Follow the rules of any CDN or cloud service you send load through, and get permission where needed. And never surprise production users or the teams who own shared systems like the database. A load test that causes an outage teaches the wrong lesson.
- double the generators: if the
- numbers change, the
- generators were the limit
- check generator CPU, ports,
- network drops
- do not load test a CDN or
- cloud service without its
- rules or permission
- never surprise production
- users or other teams
Production telemetry
The same thinking carries into production. Metrics are histograms per route, aggregated in the process and exported every few seconds. Traces are sampled: a tenth of a percent is still a thousand traces a second, and tail-based sampling keeps every error and every slow request. Logs keep errors plus a small sample. And never label metrics by user or product ID: ten million products means ten million time series, which will take down the metrics system.
- metrics: histograms per route,
- aggregated in process
- traces: sample 0.1% → still
- 1,000 traces/s at peak
- tail-based: keep all errors
- and slow ones
- logs: errors + 1% sample
- labels: never per user or per
- SKU (10M series)
SLOs and alerts
Turn the goal into a service level objective: ninety-nine point nine percent of requests succeed in under twenty-five milliseconds, measured monthly. At an average of three hundred thousand a second, that is about seven hundred and seventy-eight billion requests a month, so the error budget is around seven hundred and seventy-eight million bad ones. Do not page on every blip. Page when the budget burns, say, fourteen times faster than sustainable for an hour.
SLO: 99.9% of requests succeed in under 25 ms, per month average 300,000 req/s ≈ 777.6 billion req/month error budget 0.1% ≈ 777 million bad requests alert on burn rate: budget spent 14x too fast for 1 h → page someone
Checks in CI
Finally, do not wait for a big test to find regressions. On every build, run a ten-minute test against a single server and record requests per second per core, CPU microseconds per request, bytes allocated per request and the ninety-ninth percentile. Fail the build if any gets more than five percent worse. Remember the rule from part one: one microsecond per request is one core. A check in CI is the cheapest place to save it.
- each build: a 10-minute test
- against 1 server
- record: req/s per core, CPU µs
- per request, bytes allocated
- per request, p99
- fail the build on a > 5%
- regression
- remember: 1 µs = 1 core
- = ~$29 a month
Say it like this
In an interview: I load test with an open model at a fixed arrival rate from about twenty-five generators, measuring latency from the intended send time to avoid coordinated omission. I report merged ninety-ninth and ninety-nine point ninth percentiles, never averages. I test with realistic skewed keys, both warm and cold, ramp until I find the knee, and run production at around sixty percent of it, with SLO burn-rate alerts.
- "I test with an open model at
- a fixed rate from ~25
- generators, measuring from the
- intended send time to avoid
- coordinated omission. I report
- merged p99 and p99.9, test warm
- and cold, ramp to the knee, and
- run at ~60% of it."