SYSTEM DESIGN · DEEP DIVE · 6 PARTS · ABOUT 67 MINUTES

How to Handle 1 Million Requests per Second: A System Design Deep Dive

What really changes when one service has to answer a million HTTP requests every second? Not just the number of servers. Every microsecond of CPU becomes a core, every byte becomes bandwidth on the bill, one-in-a-million bugs fire every second, and the database has to be kept out of the way. This series follows one service, a price API for a big online sale, from the first budget to the final architecture, with the capacity math done out loud.

1 / 22

The 6 parts, in order

Each part is its own video of about ten to fourteen minutes, an animated walkthrough you can step through, written notes, and interview questions with answers.

  1. 1

    Part 1: The scale and the per-request budget

    What a million requests a second really means: one request every microsecond, the CPU budget per request, the rule that one microsecond per request costs one core, how Big-O and memory allocation turn into servers and money, Little’s Law for requests in flight, why one-in-a-million events happen every second, and the bandwidth and logging bill before any code is written.

    • What 1M rps means
    • The running example
    • The core budget
    • 1 µs = 1 core
    • Where the time goes
    • Big-O is money
    • Allocation costs
    • Little’s Law
    • Rare is constant
    • Bandwidth
    • Logs and metrics
    • What breaks first
    • Rules for scale-out
    • Latency budget
    • Say it like this

    About 11 minutes · 22 steps · 8 interview Q&As

  2. 2

    Part 2: The network edge and load balancing

    How a million requests a second reach the app servers: what a new TCP and TLS connection costs, keep-alive, HTTP/2 and HTTP/3, the load balancing tiers (GeoDNS, anycast, CDN, L4, L7) and the capacity of each, balancing algorithms, the long-lived connection trap, ephemeral port exhaustion, health checks and draining, and multiple regions.

    • The path of a request
    • Connections cost
    • Keep-alive
    • HTTP/2 and HTTP/3
    • Tier 1: DNS
    • Anycast and the CDN
    • Tier 2: L4
    • Tier 3: L7
    • Balancing algorithms
    • The long connection trap
    • Ephemeral ports
    • Health and draining
    • Reject early
    • More than one region
    • Say it like this

    About 10 minutes · 20 steps · 8 interview Q&As

  3. 3

    Part 3: The app tier: threads, event loops, CPU and memory

    Inside one app server at 40,000 requests a second: why thread-per-request caps throughput, how event loops and epoll serve many connections on few threads, Java virtual threads, never blocking the loop, kernel limits (file descriptors, accept backlog, SO_REUSEPORT, conntrack), finding CPU hot spots, serialization and compression costs, garbage collection, contention and timeouts.

    • Inside one server
    • Thread per request
    • Event loops
    • Never block the loop
    • Virtual threads
    • Choosing a model
    • Kernel limits
    • Find the CPU hot spots
    • Serialization
    • Memory and GC
    • Contention
    • Timeouts and limits
    • One server
    • Say it like this

    About 12 minutes · 22 steps · 8 interview Q&As

  4. 4

    Part 4: Caching and keeping the database out of the hot path

    Why the database must not serve a million reads a second, and how layered caches make sure it never has to: CDN micro-caching, an in-process cache, a Redis cluster, hit-rate math, hot keys, stampedes, invalidation through change events, cold starts, then the write side: batching a million view events a second through Kafka into an analytics store.

    • Why not the database
    • Layers of cache
    • Layer 1: CDN
    • Layer 2: in process
    • Layer 3: Redis
    • Hit rate math
    • Hot keys
    • Stampedes
    • Invalidation
    • Freshness budget
    • Cold starts
    • What the DB still does
    • The write side
    • Batch writers
    • What may be lost
    • Say it like this

    About 11 minutes · 20 steps · 8 interview Q&As

  5. 5

    Part 5: Load testing and measuring latency properly

    How to prove a service can take a million requests a second: generating the load from a fleet of machines, open versus closed load models, coordinated omission and how it hides slow requests, percentiles and why you never average them, realistic traffic, test shapes, finding the knee, hunting bottlenecks, testing the test, and production observability, SLOs and performance checks in CI.

    • State the goal
    • Generating 1M rps
    • Open vs closed
    • Coordinated omission
    • Percentiles
    • Realistic traffic
    • Warm and cold
    • Test shapes
    • Finding the knee
    • Watch everything
    • Bottleneck hunt
    • Test the test
    • Production telemetry
    • SLOs and alerts
    • Checks in CI
    • Say it like this

    About 11 minutes · 21 steps · 8 interview Q&As

  6. 6

    Part 6: Failure modes, cost and the final architecture

    How a million-request-a-second service fails and how to stop it: retry storms, thundering herds, hot spots, GC pauses and slow nodes, cascading failure, load shedding, circuit breakers, graceful degradation, deploys and region loss. Then the monthly bill line by line, cost per million requests, where to save, and the final architecture with every number from the series.

    • How big systems fail
    • Retry storms
    • Thundering herd
    • Hot spots
    • Slow is worse than dead
    • Cascading failure
    • Load shedding
    • Breakers and bulkheads
    • Graceful degradation
    • Changes cause outages
    • Losing a region
    • The bill
    • Where to save
    • The final architecture
    • Say it like this

    About 12 minutes · 21 steps · 8 interview Q&As

The numbers at a glance

The capacity math from the series, for quick revision. Prices and per-core rates are stated assumptions: measure your own.

1 µs per request
= 1 CPU core at 1M req/s; 1 ms = 1,000 cores Part 1
200 µs CPU, 50% busy
≈ 400 cores: 25 servers × 16 cores, 40,000 req/s each Part 1
Little’s Law
1M req/s × 20 ms = 20,000 requests in flight Part 1
2 KB responses
16 Gbit/s out; ~4.8 Gbit/s gzipped to 600 B Part 1
Keep-alive
100 requests per connection: 10,000 handshakes/s, not 1M Part 2
Ephemeral ports
~28,000 ports ÷ 60 s TIME_WAIT ≈ 470 new connections/s per backend Part 2
200 threads at 20 ms
caps a server at 10,000 req/s; event loops do not Part 3
Cache ladder
1M → CDN 50% → local 90% → Redis 99.5% → 250 DB reads/s Part 4
Every nine of hit rate
cuts database load ten times Part 4
Queueing at 90% busy
requests wait ~9 service times; run near 60% of the knee Part 5
Retries at 3 layers
4 × 4 × 4 = up to 64 attempts per request Part 6
Cost
≈ $68,000 a month, about 9 cents per million requests (assumed prices) Part 6

Questions this series answers

How many servers do you need for 1 million requests per second?

Measure CPU time per request, then cores = requests per second × CPU seconds per request ÷ target utilisation. At 200 microseconds and 50% target that is 400 cores, for example 25 servers with 16 cores, each serving 40,000 requests a second.

Why does one microsecond per request matter at this scale?

A million requests a second times one microsecond is one second of CPU every second, which is one full core. One extra millisecond per request therefore needs about a thousand more cores, a large monthly cost.

What is the difference between L4 and L7 load balancing?

L4 forwards packets by IP address and port without reading the request, so it is cheap and handles millions of packets per second. L7 terminates TLS and parses HTTP, so it can route by path or header and balance each request, but it costs CPU on every request.

Why is keep-alive essential at 1 million requests per second?

Each new connection costs TCP and TLS round trips and public-key CPU work. Reusing a connection for about 100 requests turns a million handshakes per second into ten thousand, saving hundreds of cores and user latency.

Why does thread-per-request struggle at high request rates?

Throughput is capped at threads divided by latency, so 200 threads at 20 ms give only 10,000 requests per second, and far less when a dependency slows. Adding thousands of OS threads costs memory and context switching.

How does an event loop handle many connections with few threads?

Each loop thread asks the kernel (epoll) which sockets are ready, handles them without blocking, and registers callbacks for outbound I/O. One thread per core can serve tens of thousands of connections because no thread waits idle.

Why should the database not serve 1 million reads per second directly?

It would need dozens of replicas, each replaying every write, plus thousands of connections, and any slow query would hurt every request. Layered caches can absorb almost all reads at a fraction of the cost.

How do you calculate the load that reaches the database through several caches?

Multiply the miss rates. For example 1,000,000 requests with a 50% CDN hit rate, a 90% in-process hit rate and a 99.5% Redis hit rate leave 1,000,000 × 0.5 × 0.1 × 0.005 = 250 reads per second for the database.

How do you generate 1 million requests per second for a load test?

Use a fleet of load generators, for example 25 machines each sending 40,000 requests per second at a fixed arrival rate, coordinated by a controller that merges their latency histograms. Keep each generator below about 60% CPU and raise its port and file limits.

What is coordinated omission?

When a closed-loop load tool waits for slow responses, it stops sending during a stall, so the requests that should have been sent are never measured. Percentiles then look far better than reality. Measure latency from the intended send time with a constant-rate tool.

What is a retry storm and how do you prevent it?

When several layers each retry failed calls, attempts multiply, for example 4 × 4 × 4 = 64 for three layers with three retries, overloading a struggling dependency. Retry at one layer, cap retries with a budget, use exponential backoff with jitter, and never retry overload responses.

What is a thundering herd?

Many clients acting at the same instant, such as reconnecting after an outage, timers firing on the hour, or cache entries expiring together. Spread actions with random delays, cap admission at the edge, make reconnects cheap with session resumption and jitter expiry times.

Related walkthroughs

Learn it, then keep it

Mark each part as watched in the app, and it comes back for two quick revisions of the notes and interview questions.

Open in the app →