SYSTEM DESIGN · DEEP DIVE · 1M REQUESTS PER SECOND · PART 4 OF 6

Handling 1 Million Requests per Second, Part 4: Caching and Keeping the Database Out of the Hot Path

Why the database must not serve a million reads a second, and how layered caches make sure it never has to: CDN micro-caching, an in-process cache, a Redis cluster, hit-rate math, hot keys, stampedes, invalidation through change events, cold starts, then the write side: batching a million view events a second through Kafka into an analytics store.

Part 4 of 6 in How to Handle 1 Million Requests per Second, a deep dive in 6 videos.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
1 / 20

Press play to hear the narration, or step through at your own pace.

Notes

Why not the database

First, why not just use the database? Assume one replica serves around thirty thousand indexed lookups a second. Measure yours. A million reads needs thirty-four replicas, and every one replays every write. Twenty-five servers with fifty connections each is over a thousand connections per database, and in Postgres each is a process. One bad query plan or maintenance spike, and every request in the company slows down at once.

Compare the cost of one read at each layer. An in-process cache is a hash lookup in local memory, around a hundred nanoseconds. Redis is one network round trip and a hash lookup, well under a millisecond, and a single shard handles a hundred thousand or more operations a second. A database parses SQL, plans, walks an index, checks row versions for isolation, then replies. Each step further from the CPU costs roughly ten times more, so serve reads as high up as possible.

Straight to the database?
assume one replica serves
  ~30,000 indexed lookups/s
1,000,000 ÷ 30,000 = 34 replicas
  each replaying every write

connections: 25 servers × 50
  = 1,250 per database
  Postgres: a process each

one slow plan or vacuum spike
  → every request is slow
Cost per read
  • in-process cache: a hash lookup
  • in local RAM, ~100 ns
  • Redis: one network round trip,
  • ~0.3–1 ms, ~100k+ ops/s per
  • shard
  • database: parse SQL, plan, walk
  • a B-tree, check row versions,
  • then a round trip
  • each layer down costs ~10x more

Layers of cache

Here is the plan as a ladder. A million requests reach the CDN. If it answers half of them, five hundred thousand reach our servers. Each server’s in-process cache answers ninety percent, so fifty thousand go to Redis. Redis answers ninety-nine and a half percent, so about two hundred and fifty a second reach the database: one read in four thousand. These hit rates are assumptions to measure. The design works only while they hold, so alert when any of them drops.

The hit-rate ladder
users          1,000,000 req/s
CDN hit 50%  →   500,000 reach us
in-process 90% →  50,000 to Redis
Redis 99.5%   →     250 to the DB

the DB sees 1 in 4,000 reads

assumed hit rates: measure them
per layer and alert on drops

Layer 1: CDN

Layer one is the CDN. Prices may be shown a couple of seconds stale, so the response says the edge may cache it for two seconds, and serve a stale copy while it fetches a fresh one. The cache key is product and region. A hot product getting fifty thousand requests a second, spread across about four hundred edge sites, now costs us one fetch per site every two seconds, around two hundred a second. Anything personal, like a member discount, goes in a separate uncached call.

Micro-caching at the edge
Cache-Control: public,
  s-maxage=2,
  stale-while-revalidate=10
cache key: sku + region

hot SKU at 50,000 req/s
  ~400 edge sites, 1 fetch
  per site per 2 s
  → ~200 req/s reach us
personal discounts: separate
  uncached call

Layer 2: in process

Layer two lives inside each app server: a bounded in-memory cache such as Caffeine. The hot set, maybe fifty thousand product and region pairs holding the pre-encoded bytes from part three, is about a hundred and twenty-five megabytes. Entries expire after a second and refresh just before. The catch is twenty-five independent copies that can disagree for up to a second, so when a price changes we also broadcast a delete to every server.

In-process cache (Caffeine)
hot set: ~50,000 sku+region
  × 2.5 KB pre-encoded bytes
  ≈ 125 MB per server
expire after 1 s, refresh
  ahead of expiry
bounded by size: W-TinyLFU

25 servers = 25 copies:
  can disagree for up to 1 s
broadcast deletes on change

Layer 3: Redis

Layer three is a Redis cluster shared by all servers. In normal running it sees fifty thousand operations a second, which one node could handle. But size it for the bad day, when local caches are empty after a deploy and Redis briefly sees all five hundred thousand. Assuming a hundred thousand a second per shard, that is six shards, each with a replica. Ten million products in five regions at two and a half kilobytes is about a hundred and twenty-five gigabytes, twenty-one per shard.

Redis cluster sizing
normal: 50,000 ops/s
local caches cold (deploy,
  restart): up to 500,000/s
assume 100,000 ops/s per shard
  → 6 shards, 1 replica each

data: 10M SKUs × 5 regions
  × 2.5 KB ≈ 125 GB
  ≈ 21 GB per shard

pipelined, multiplexed clients

Hit rate math

Here is why hit rate is the number to watch. Of the five hundred thousand reads reaching our servers, a ninety percent combined hit rate sends fifty thousand to the database. Ninety-nine percent sends five thousand. Ninety-nine point nine sends five hundred. Each extra nine is a tenfold cut. Turn it around: a small drop in hit rate, from ninety-nine point nine five to ninety-nine, multiplies database load by twenty. That is how a cache problem becomes a database outage.

Every nine is a 10x cut
500,000 reads/s reach origin

combined hit   DB reads/s
  90%            50,000
  99%             5,000
  99.9%             500
  99.95%            250

a 1% drop in hit rate can
multiply DB load by 20

Hot keys

Sharding spreads keys, not load. When one product goes into a flash deal and gets two hundred thousand requests a second, all of them hash to the same Redis shard, which can do maybe half that. The in-process cache is the main defence, because each server asks Redis at most once a second for that key. Then the CDN. For extra safety, store eight copies of the hot key with suffixes and read a random one. And detect hot keys by sampling the most requested keys in the client.

One key, one shard
  • flash deal on one SKU:
  • 200,000 req/s for 1 key
  • one shard handles ~100,000
  • fixes, in order:
  • in-process cache absorbs it
  • CDN micro-cache
  • copies: price:42#0 … #7,
  • read a random copy
  • read from shard replicas
  • detect: client-side top-K

Stampedes

The next failure is the stampede. A hot key expires, thousands of requests miss at the same moment, and every one of them queries the database for the same row. The fix is single flight: per server, only the first miss loads the value, and the rest wait for that one result, so at most one load per server. Better still, refresh popular keys before they expire, serve the stale value while refreshing, and add random jitter to expiry times so keys do not all expire together.

Cache stampede
  • hot key expires; 9,000 requests
  • miss at once; all query the
  • DB for the same row
  • single flight: 1 load per key
  • per server, others wait for
  • it → at most 25 loads
  • refresh ahead before expiry
  • serve stale while refreshing
  • random TTL jitter, ±20%
  • Redis SET NX lock for 1 loader

Invalidation

How does a new price reach the caches? The pricing service writes to the database. A change feed, from the database log or an outbox table, publishes an event to Kafka. An updater rebuilds the encoded response, writes it to Redis in every region, and broadcasts a delete so each server drops its local copy. The CDN copy expires within two seconds. So caches are refreshed by events, and the time-to-live is only a safety net. Worst-case staleness: a few seconds.

Freshness budget

Caching is a product decision as much as a technical one, so write down a freshness budget. A price on a product tile may be two or three seconds stale. The in-stock flag, five seconds. The price at checkout must be correct, so checkout reads the database, or the order carries the price version it showed, and the server verifies it. Personal discounts are never cached. Stating this up front saves arguments later.

Write down the freshness
  • price on a product tile:
  • up to 2–3 s stale is fine
  • stock flag: up to 5 s
  • price at checkout: read from
  • the database, or check the
  • version in the order
  • personal discount: no cache
  • state it in the design, not
  • after the first complaint

Cold starts

Caches create a new failure: the cold start. If Redis loses a shard, or someone flushes the cache, every request misses and the database receives the full load it was never sized for. So never flush everything at peak. Warm caches before they take traffic by preloading the hottest keys. Give each Redis shard a replica so a failover keeps the data. Let new servers ramp up slowly. And cap database concurrency, so misses wait in line instead of crushing it.

Avoiding a cold cache
  • never flush the whole cache
  • at peak
  • warm before taking traffic:
  • preload the top 1M keys
  • Redis: replicas, so a failover
  • keeps the data
  • new app servers: slow start
  • (part 2) while local caches
  • fill
  • DB: a concurrency cap, so
  • misses queue, not pile up

What the DB still does

The database still matters. It is the source of truth: two thousand price changes a second, a few hundred to a few thousand reads a second from cache misses, and the checkout reads. That is a modest load: one primary and two replicas for failover and misses. A connection pooler in front caps real connections at about fifty, however many app servers we add. Compare that with the thirty-four replicas we would need without the caches.

The database, right-sized
writes: 2,000 price changes/s
reads: ~250–5,000/s of misses
  plus checkout reads

1 primary + 2 replicas
  (failover, misses, checkout)
connection pooler: 25 servers
  × 10 → ~50 real connections

vs 34 replicas without caches

The write side

Now the write side. Every request records a view event, so that is a million events a second. Inserting each one into a database is out of the question. Instead, each app server hands events to an asynchronous Kafka producer that groups them into compressed batches of around sixty-four kilobytes every few milliseconds. The request never waits for Kafka’s acknowledgement, and the producer buffer has a fixed size, so a slow Kafka cannot fill the server’s memory.

Size Kafka with the same arithmetic. A million events of a hundred and fifty bytes is a hundred and fifty megabytes a second. Compression shrinks that about four times, to forty. With three copies of every partition, the brokers write a hundred and twenty megabytes a second in total. Forty-eight partitions let many consumers work in parallel. Keeping a day of events is about three and a half terabytes, ten with replication. Six brokers handle it comfortably.

View events
1 view event per request
  = 1,000,000 events/s

one INSERT each: impossible

app: async producer, batches
  of ~64 KB every few ms
  (linger.ms=5, zstd)
never wait for the ack on the
  hot path; bounded buffer
Kafka sizing
1,000,000 × 150 B = 150 MB/s
compressed ~4x → ~40 MB/s
replication factor 3
  → ~120 MB/s of disk writes
  across the brokers
48 partitions for parallel
  consumers, 6 brokers
keep 24 h: 40 MB/s × 86,400
  ≈ 3.5 TB × 3 ≈ 10 TB of disk

Batch writers

Batch writers consume the events. They sum views per product per minute in memory, then write large batches, around a hundred thousand rows at a time, into an analytics store such as ClickHouse, which is built for exactly that. They commit Kafka offsets only after a successful write, and use event IDs to ignore duplicates. Raw events are also archived to object storage for replays. Our transactional database never sees a single view event.

From events to rows
  • consumers sum views per SKU
  • per minute in memory
  • write to an analytics store
  • (ClickHouse): inserts of
  • ~100,000 rows at a time
  • commit Kafka offsets after the
  • write; dedupe by event id
  • raw events also land in object
  • storage for replays
  • the OLTP DB never sees them

What may be lost

Last, decide what can be lost. View events are analytics. If Kafka is slow and the producer buffer fills, drop them and count how many you dropped. A price read must never fail because analytics is having a bad day. Price changes are different: they are written to the database first and are never lost. Orders do not touch this path at all. Making this explicit is what lets the hot path stay simple and fast.

Loss budget
  • view events: analytics, may
  • drop under overload, count
  • what you dropped
  • price changes: never lost,
  • written to the DB first
  • orders: never on this path
  • a price read must not fail
  • because Kafka is slow

Say it like this

In an interview: reads go through a two-second CDN micro-cache, an in-process cache, then a six-shard Redis cluster, so the database sees about one read in four thousand. Price changes flow out as events that update every cache layer. I protect against hot keys with local caching and key copies, and against stampedes with single flight and refresh ahead. View events go through Kafka in batches into ClickHouse, never into the main database.

In the interview
  • "Reads: CDN micro-cache for 2 s,
  • an in-process cache, then a
  • 6-shard Redis cluster, so the
  • DB sees ~1 in 4,000 reads.
  • Changes flow by events. I guard
  • hot keys and stampedes. Views
  • go through Kafka in batches
  • into ClickHouse."

Interview questions

Why should the database not serve 1 million reads per second directly?

It would need dozens of replicas, each replaying every write, plus thousands of connections, and any slow query would hurt every request. Layered caches can absorb almost all reads at a fraction of the cost.

How do you calculate the load that reaches the database through several caches?

Multiply the miss rates. For example 1,000,000 requests with a 50% CDN hit rate, a 90% in-process hit rate and a 99.5% Redis hit rate leave 1,000,000 × 0.5 × 0.1 × 0.005 = 250 reads per second for the database.

What is micro-caching at the CDN?

Caching dynamic responses for a very short time, such as 1 to 2 seconds, keyed by the parts of the request that change the answer. A hot item then costs one origin fetch per edge site per period instead of every request.

How do you handle a hot key that one Redis shard cannot serve?

Absorb it in an in-process cache and at the CDN, store several copies of the key with suffixes and read a random copy, or read from shard replicas. Detect hot keys by sampling top requested keys on the client.

What is a cache stampede and how do you prevent it?

Many requests miss the same expired key at once and all load it from the database. Prevent it with single-flight loading per key, refresh-ahead, serving stale data while refreshing, and jittered expiry times.

How do you keep caches up to date when prices change?

Write the change to the database, publish it through a change feed or outbox to a queue, rebuild the cached value in Redis, and broadcast deletes to in-process caches. Expiry times remain as a safety net.

How do you handle a million analytics events per second?

Produce them asynchronously in compressed batches to Kafka, consume and aggregate them, and write large batches to an analytics store such as ClickHouse. Commit offsets after writing and deduplicate by event ID.

What is a cache cold start and why is it dangerous?

An empty cache, after a flush, failover or deploy, sends every read to the database at once. Warm caches before traffic, keep replicas of cache shards, ramp new servers slowly and cap database concurrency.