SYSTEM DESIGN · DEEP DIVE · 1M REQUESTS PER SECOND · PART 6 OF 6

Handling 1 Million Requests per Second, Part 6: Failure Modes, Cost and the Final Architecture

How a million-request-a-second service fails and how to stop it: retry storms, thundering herds, hot spots, GC pauses and slow nodes, cascading failure, load shedding, circuit breakers, graceful degradation, deploys and region loss. Then the monthly bill line by line, cost per million requests, where to save, and the final architecture with every number from the series.

Part 6 of 6 in How to Handle 1 Million Requests per Second, a deep dive in 6 videos.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
1 / 21

Press play to hear the narration, or step through at your own pace.

Notes

How big systems fail

Here is the list of amplifiers to design against. Retries multiply load exactly when the system is weakest. Herds of clients arrive at the same instant. Hot spots concentrate load on one shard. Slow nodes hold on to requests. Slowness grows the amount of work in flight, as Little’s Law told us. And most incidents start with a change: a deploy or a config push. Each amplifier needs a damper, chosen calmly before the incident, not during it.

Amplifiers
  • retries multiply load
  • herds arrive all at once
  • hot spots concentrate load
  • slow nodes hold requests
  • slowness grows in-flight work
  • (Little’s Law)
  • changes: deploys and config
  • each needs a damper, chosen
  • before the incident

Retry storms

Retry storms first. If the client, the proxy and the app server each try once and retry three times, one failing request at the bottom can be attempted four times four times four: sixty-four times. Suppose Redis blips for two seconds and ten percent of calls fail. With retries at every layer, its normal fifty thousand operations a second can become several hundred thousand, far above its capacity, so the two-second blip becomes a real outage.

The dampers. Retry at one layer only, usually the one nearest the user. Give retries a budget: no more than about ten percent extra requests, and stop retrying once the budget is spent. Back off exponentially with full jitter, meaning a random wait up to a growing cap, so retries do not arrive in waves. Retry only idempotent calls. Never retry an overload response. And pass a deadline down with each request, so no layer retries work that the caller has already abandoned.

Retries multiply
each layer: 1 try + 3 retries
  = 4 attempts
client → L7 → app → Redis
  4 × 4 × 4 = 64 attempts
  for one user request

Redis blips for 2 s, 10% fail
  normal 50,000 ops/s
  → several hundred thousand
  ops/s of retries
  → the blip becomes an outage
Damping retries
  • retry at one layer only,
  • usually nearest the user
  • retry budget: retries ≤ 10%
  • of requests, else stop
  • exponential backoff + full
  • jitter: sleep random(0, cap)
  • retry only idempotent calls
  • never retry 503 overload or
  • 429; honour Retry-After
  • propagate deadlines: no retry
  • after the caller gave up

Thundering herd

Next, the thundering herd. After a thirty-second outage, two million clients reconnect in the same second. Each full TLS handshake costs about a tenth of a millisecond, so that is two hundred core-seconds of work: every proxy core in all three regions flat out for more than a second, with real requests waiting behind it. Herds also come from apps that refresh on the hour, push campaigns that open millions of apps at once, and cache entries all set with the same expiry.

To tame herds, spread the return over time and make returning cheap. Client libraries reconnect after a random delay, spread over thirty to sixty seconds. TLS session tickets let returning clients skip the costly handshake. The edge admits new connections at a capped rate while serving existing ones. Add random jitter to every timer and cache expiry, stagger push notification campaigns, and warm the caches before you let traffic back in.

The herd
30 s outage, then recovery
2,000,000 clients reconnect in
  the same second
full TLS handshake ≈ 0.1 ms
  → 200 core-seconds of work
  → all ~190 proxy cores at
  100% for over 1 s, real
  traffic waits behind it

also: timers at :00, push
  campaigns, TTLs set together
Taming herds
  • clients: reconnect with random
  • delay, spread over 30–60 s
  • session tickets: resumptions
  • skip the costly handshake
  • edge: admission control,
  • accept new connections at
  • a capped rate
  • jitter every timer and TTL
  • stagger push campaigns
  • warm caches before reopening

Hot spots

Hot spots go beyond the hot cache key from part four. If view events are keyed by product, a flash deal sends a third of them into one Kafka partition and its consumer falls behind. Views do not need ordering, so spread them randomly. A single partner sending thirty percent of all traffic needs its own rate limit. The general habit: for every sharded thing, graph the busiest shard against the median shard, and alert when they drift apart.

Hot spots beyond the cache
  • Kafka keyed by SKU: a flash
  • deal fills one partition;
  • key views randomly instead,
  • order is not needed
  • one tenant or API key at 30%
  • of traffic: per-key limits
  • one region’s prices popular
  • worldwide: replicate them
  • look for skew: max vs median
  • per shard, per partition

Slow is worse than dead

A slow server is worse than a dead one. A dead server fails health checks and is removed in seconds. A server stuck in a three hundred millisecond garbage collection pause, or with a failing disk, keeps accepting work: at forty thousand requests a second, twelve thousand requests pile up behind one pause. So proxies should eject servers whose latency is an outlier, not only those returning errors. For reads, hedging helps: if no reply arrives by the ninety-fifth percentile, send a second copy elsewhere.

Grey failures
one server pauses 300 ms
  × 40,000 req/s = 12,000
  requests stuck behind it
a dead server is ejected in
  seconds; a slow one is not

outlier ejection on latency,
  not only on errors
hedged reads: if no reply by
  p95, ask a second server
  (≈5% extra load)

Cascading failure

Put those together and you get a cascading failure. A dependency slows from twenty to five hundred milliseconds. Requests in flight per server jump from eight hundred to twenty thousand. Memory rises, garbage collection pauses get longer, health checks time out, and servers are ejected. The survivors receive even more traffic, so they slow down too, while clients retry. Within a minute, a slow dependency has become a total outage. The cure is to refuse work early.

The death spiral
  • dependency slows 20 → 500 ms
  • in flight per server 800 →
  • 20,000 (Little’s Law)
  • memory up → longer GC pauses
  • health checks time out →
  • servers ejected
  • survivors get more traffic →
  • they slow down too
  • clients retry → more load
  • result: total outage

Load shedding

Load shedding is that cure. When requests in flight pass a limit, or time spent queued passes twenty milliseconds, the server answers immediately with a five oh three. A fast refusal costs a few microseconds; a slow success holds memory and ends in a timeout anyway. Shed by priority: analytics and prefetches first, product tiles next, the cart and checkout last. Adaptive concurrency limits, which raise and lower the cap based on measured latency much like TCP does, can tune this automatically.

Shed load early
reject when: in flight > 4,000
  or queue wait > 20 ms
a fast 503 costs ~5 µs; a
  slow success costs memory,
  a thread and a timeout

shed by priority:
  1 analytics, prefetch
  2 product tiles
  3 cart and checkout: last
adaptive limits (like TCP)

Breakers and bulkheads

Two more dampers. A circuit breaker per dependency watches errors: if half the calls fail over ten seconds, it opens, and calls fail fast to a fallback instead of waiting. For prices, the fallback is the last known value from the local cache, marked as possibly stale. After a pause the breaker lets a few probes through and closes if they succeed. Bulkheads give each dependency its own pool and limit, so a slow delivery-estimate service cannot use up capacity the price lookup needs.

Circuit breakers, bulkheads
  • breaker per dependency: 50%
  • errors over 10 s → open:
  • fail fast, use a fallback
  • → half-open: a few probes
  • fallback for prices: last known
  • value from the local cache,
  • flagged as possibly stale
  • bulkheads: separate pools and
  • limits per dependency, so
  • delivery estimates cannot
  • eat the price lane

Graceful degradation

Plan graceful degradation as an ordered list. Step one, drop analytics view events and count them. Step two, skip the expensive delivery estimate. Step three, serve prices up to a minute stale from local caches if Redis is in trouble. Step four, let the CDN serve stale responses when our origin errors. Throughout, checkout keeps verifying the price against the database, so stale data can never cost the business money. Deciding this order in advance turns a crisis into a checklist.

Degrade in steps
  • 1 drop view events (counted)
  • 2 skip delivery estimates
  • 3 serve prices up to 60 s
  • stale from local cache
  • 4 CDN serves stale on error:
  • stale-if-error=300
  • 5 checkout still verifies
  • price in the database
  • decide the order in advance

Changes cause outages

Most outages start with a change, so changes need the same care as traffic. Every deploy starts as a canary on one server, compared for ten minutes against the rest on latency, errors and CPU per request. Then roll three servers at a time. With twenty-two of twenty-five serving, each takes forty-five thousand requests a second at about fifty-six percent busy, which our headroom covers. Config pushes are canaried too, rollbacks are automatic, and nothing big ships at peak.

Safe changes
canary: 1 server, ~4% of a
  region, for 10 min
  compare p99, errors, CPU µs
  per request vs the rest
then roll 3 servers at a time
  22 carry 1M: 45,000 each,
  ~56% busy ✓
config pushes: canaried too
auto rollback on SLO burn
no big changes at peak

Losing a region

Losing a region tests everything at once. With three regions each sized for half a million, capacity is fine. The trap is caches. The surviving regions’ caches are full of their own users’ keys, so the moved traffic misses, and database load spikes exactly when you can least afford it. The fix: the price updater from part four writes every region’s keys into every region’s Redis, so failover lands on warm caches. And practise it with planned game days, not for the first time during a real outage.

Region failover
  • 3 regions, each sized for 500k
  • one fails: DNS and anycast
  • move its users in minutes
  • trap: the survivors’ caches
  • hold their own regions’
  • keys, misses spike
  • fix: the price updater writes
  • every region’s keys into
  • every region’s Redis
  • practice it: game days

The bill

Now the bill, with assumed prices to replace with your own. Six hundred app cores in three regions at twenty-nine dollars each: about seventeen thousand a month. Proxies five and a half thousand, Redis nine, Kafka and ClickHouse eight, databases three. Bandwidth: compressed responses at the average rate are about four hundred and sixty-seven terabytes a month, around twenty-three thousand dollars. Total: about sixty-eight thousand. Unsampled logs, at fifty cents a gigabyte, would add a hundred and fifty thousand.

Turn the bill into one number: cost per million requests. At an average of three hundred thousand a second, we serve about seven hundred and seventy-eight billion requests a month. Sixty-eight thousand dollars divided by that is roughly nine cents per million requests. Track it next to latency, per release. If it rises, something got slower, bigger or chattier, and you can find it before the invoice does.

Monthly bill (assumed prices)
app: 600 cores × $29     $17,400
L7 proxies: 192 cores     $5,600
Redis: 36 nodes × $250    $9,000
Kafka + ClickHouse        $8,000
database, 3 regions       $3,000
egress: 467 TB × $0.05   $23,300
logs, sampled             $2,000
                        --------
total                ≈ $68,000
logs unsampled at $0.50/GB:
  +$154,000
Cost per million requests
average 300,000 req/s
  ≈ 777.6 billion req/month

$68,000 ÷ 777,600 million
  ≈ $0.09 per million requests

track it like latency:
  per build, per release
if it rises, something got
  slower or bigger

Where to save

Where do the savings come from? Bytes first: compress, remove fields clients do not use, and cache at the edge. Then logs: sample, but keep every error. Then CPU per request, with the tricks from part three. Scale the stateless fleet with the daily curve, since the average is only thirty percent of the peak, and run it on cheaper machine types or committed pricing. But do not save on headroom, regions or replicas. Those are what you are really paying for.

Where to save
  • bytes: compress, trim fields,
  • cache at the edge
  • logs: sample, keep errors
  • CPU: encode once, cut allocs
  • scale with the daily curve:
  • average is 30% of the peak
  • cheaper machines for stateless
  • servers: ARM, spot, commits
  • do not save on: headroom,
  • regions, replicas

The final architecture

Here is the final architecture. Users reach an anycast CDN that finishes TLS and micro-caches prices for two seconds, answering half the traffic. The rest flows through layer-four and layer-seven balancers to event-loop app servers, which answer ninety percent from their local cache and most of the remainder from Redis. The database is the source of truth, feeding price changes back out as events. View events go through Kafka into ClickHouse. Telemetry watches all of it, in three active regions.

And the numbers, per region, sized for half a million requests a second. About half answered by the CDN. Four layer-four boxes and eight eight-core proxies. Thirteen sixteen-core app servers plus spares, at two hundred microseconds of CPU per request and no more than half busy. Local caches at ninety percent, Redis at ninety-nine and a half. Six Redis shards with replicas, a small database, six Kafka brokers. A ninety-ninth percentile under twenty-five milliseconds, at about nine cents per million requests.

The numbers, per region
sized for 500,000 req/s
CDN micro-cache: ~50% hit
L4: 4 boxes · L7: 8 × 8 cores
app: ~13 × 16 cores (+ spares)
  200 µs CPU, ≤50% busy
local cache 90% · Redis 99.5%
Redis: 6 shards + 6 replicas
DB: primary + 2 replicas
Kafka: 6 brokers, 48 parts
p99 under 25 ms · 9¢ per 1M

Say it like this

In an interview, close with risks and cost. The biggest risks are amplifiers, so I retry at one layer only, with a budget and jitter. I shed load early by priority, use circuit breakers with stale fallbacks, and canary every deploy and config change. Three active regions are each sized for half a million requests a second, with caches pre-warmed for failover. And it costs about nine cents per million requests, which I track per release.

In the interview
  • "Biggest risks are amplifiers:
  • I retry at one layer with a
  • budget and jitter, shed load
  • early by priority, use breakers
  • with stale fallbacks, canary
  • every change, and size 3
  • regions for 500k each. It costs
  • ~9¢ per million requests."

Interview questions

What is a retry storm and how do you prevent it?

When several layers each retry failed calls, attempts multiply, for example 4 × 4 × 4 = 64 for three layers with three retries, overloading a struggling dependency. Retry at one layer, cap retries with a budget, use exponential backoff with jitter, and never retry overload responses.

What is a thundering herd?

Many clients acting at the same instant, such as reconnecting after an outage, timers firing on the hour, or cache entries expiring together. Spread actions with random delays, cap admission at the edge, make reconnects cheap with session resumption and jitter expiry times.

What is load shedding and why does it help?

Rejecting requests quickly, for example with a 503, when in-flight work or queue time passes a limit. A fast rejection costs microseconds, while accepting more than you can serve grows memory and latency until everything times out.

Why is a slow server worse than a dead one?

A dead server fails health checks and is removed quickly, while a slow one keeps accepting requests that wait behind it, raising tail latency for everyone. Eject latency outliers and use hedged requests for idempotent reads.

How does a cascading failure happen?

A slow dependency raises in-flight requests by Little’s Law, which raises memory and GC pauses, which fail health checks, which remove servers and push more load onto the survivors, while retries add even more. Timeouts, load shedding, breakers and retry budgets break the loop.

How do you estimate the monthly cost of a 1 million requests per second service?

Add compute (cores × price), cache and queue clusters, databases, egress (average bytes per second × seconds per month × price per GB) and observability. Then divide by monthly requests to get cost per million requests and track it per release.

What usually dominates cost at this scale?

Often bandwidth and observability rather than servers. Compressing payloads, caching at the edge and sampling logs can save more than optimising compute, though CPU per request still matters.

How do you make a region failover safe?

Size each region to carry its share if one fails, replicate data and cache contents to every region so failover lands on warm caches, shift traffic with DNS or anycast, and rehearse it with game days.