Notes
Why not the database
First, why not just use the database? Assume one replica serves around thirty thousand indexed lookups a second. Measure yours. A million reads needs thirty-four replicas, and every one replays every write. Twenty-five servers with fifty connections each is over a thousand connections per database, and in Postgres each is a process. One bad query plan or maintenance spike, and every request in the company slows down at once.
Compare the cost of one read at each layer. An in-process cache is a hash lookup in local memory, around a hundred nanoseconds. Redis is one network round trip and a hash lookup, well under a millisecond, and a single shard handles a hundred thousand or more operations a second. A database parses SQL, plans, walks an index, checks row versions for isolation, then replies. Each step further from the CPU costs roughly ten times more, so serve reads as high up as possible.
assume one replica serves ~30,000 indexed lookups/s 1,000,000 ÷ 30,000 = 34 replicas each replaying every write connections: 25 servers × 50 = 1,250 per database Postgres: a process each one slow plan or vacuum spike → every request is slow
- in-process cache: a hash lookup
- in local RAM, ~100 ns
- Redis: one network round trip,
- ~0.3–1 ms, ~100k+ ops/s per
- shard
- database: parse SQL, plan, walk
- a B-tree, check row versions,
- then a round trip
- each layer down costs ~10x more
Layers of cache
Here is the plan as a ladder. A million requests reach the CDN. If it answers half of them, five hundred thousand reach our servers. Each server’s in-process cache answers ninety percent, so fifty thousand go to Redis. Redis answers ninety-nine and a half percent, so about two hundred and fifty a second reach the database: one read in four thousand. These hit rates are assumptions to measure. The design works only while they hold, so alert when any of them drops.
users 1,000,000 req/s CDN hit 50% → 500,000 reach us in-process 90% → 50,000 to Redis Redis 99.5% → 250 to the DB the DB sees 1 in 4,000 reads assumed hit rates: measure them per layer and alert on drops
Layer 1: CDN
Layer one is the CDN. Prices may be shown a couple of seconds stale, so the response says the edge may cache it for two seconds, and serve a stale copy while it fetches a fresh one. The cache key is product and region. A hot product getting fifty thousand requests a second, spread across about four hundred edge sites, now costs us one fetch per site every two seconds, around two hundred a second. Anything personal, like a member discount, goes in a separate uncached call.
Cache-Control: public, s-maxage=2, stale-while-revalidate=10 cache key: sku + region hot SKU at 50,000 req/s ~400 edge sites, 1 fetch per site per 2 s → ~200 req/s reach us personal discounts: separate uncached call
Layer 2: in process
Layer two lives inside each app server: a bounded in-memory cache such as Caffeine. The hot set, maybe fifty thousand product and region pairs holding the pre-encoded bytes from part three, is about a hundred and twenty-five megabytes. Entries expire after a second and refresh just before. The catch is twenty-five independent copies that can disagree for up to a second, so when a price changes we also broadcast a delete to every server.
hot set: ~50,000 sku+region × 2.5 KB pre-encoded bytes ≈ 125 MB per server expire after 1 s, refresh ahead of expiry bounded by size: W-TinyLFU 25 servers = 25 copies: can disagree for up to 1 s broadcast deletes on change
Layer 3: Redis
Layer three is a Redis cluster shared by all servers. In normal running it sees fifty thousand operations a second, which one node could handle. But size it for the bad day, when local caches are empty after a deploy and Redis briefly sees all five hundred thousand. Assuming a hundred thousand a second per shard, that is six shards, each with a replica. Ten million products in five regions at two and a half kilobytes is about a hundred and twenty-five gigabytes, twenty-one per shard.
normal: 50,000 ops/s local caches cold (deploy, restart): up to 500,000/s assume 100,000 ops/s per shard → 6 shards, 1 replica each data: 10M SKUs × 5 regions × 2.5 KB ≈ 125 GB ≈ 21 GB per shard pipelined, multiplexed clients
Hit rate math
Here is why hit rate is the number to watch. Of the five hundred thousand reads reaching our servers, a ninety percent combined hit rate sends fifty thousand to the database. Ninety-nine percent sends five thousand. Ninety-nine point nine sends five hundred. Each extra nine is a tenfold cut. Turn it around: a small drop in hit rate, from ninety-nine point nine five to ninety-nine, multiplies database load by twenty. That is how a cache problem becomes a database outage.
500,000 reads/s reach origin combined hit DB reads/s 90% 50,000 99% 5,000 99.9% 500 99.95% 250 a 1% drop in hit rate can multiply DB load by 20
Hot keys
Sharding spreads keys, not load. When one product goes into a flash deal and gets two hundred thousand requests a second, all of them hash to the same Redis shard, which can do maybe half that. The in-process cache is the main defence, because each server asks Redis at most once a second for that key. Then the CDN. For extra safety, store eight copies of the hot key with suffixes and read a random one. And detect hot keys by sampling the most requested keys in the client.
- flash deal on one SKU:
- 200,000 req/s for 1 key
- one shard handles ~100,000
- fixes, in order:
- in-process cache absorbs it
- CDN micro-cache
- copies: price:42#0 … #7,
- read a random copy
- read from shard replicas
- detect: client-side top-K
Stampedes
The next failure is the stampede. A hot key expires, thousands of requests miss at the same moment, and every one of them queries the database for the same row. The fix is single flight: per server, only the first miss loads the value, and the rest wait for that one result, so at most one load per server. Better still, refresh popular keys before they expire, serve the stale value while refreshing, and add random jitter to expiry times so keys do not all expire together.
- hot key expires; 9,000 requests
- miss at once; all query the
- DB for the same row
- single flight: 1 load per key
- per server, others wait for
- it → at most 25 loads
- refresh ahead before expiry
- serve stale while refreshing
- random TTL jitter, ±20%
- Redis SET NX lock for 1 loader
Invalidation
How does a new price reach the caches? The pricing service writes to the database. A change feed, from the database log or an outbox table, publishes an event to Kafka. An updater rebuilds the encoded response, writes it to Redis in every region, and broadcasts a delete so each server drops its local copy. The CDN copy expires within two seconds. So caches are refreshed by events, and the time-to-live is only a safety net. Worst-case staleness: a few seconds.
Freshness budget
Caching is a product decision as much as a technical one, so write down a freshness budget. A price on a product tile may be two or three seconds stale. The in-stock flag, five seconds. The price at checkout must be correct, so checkout reads the database, or the order carries the price version it showed, and the server verifies it. Personal discounts are never cached. Stating this up front saves arguments later.
- price on a product tile:
- up to 2–3 s stale is fine
- stock flag: up to 5 s
- price at checkout: read from
- the database, or check the
- version in the order
- personal discount: no cache
- state it in the design, not
- after the first complaint
Cold starts
Caches create a new failure: the cold start. If Redis loses a shard, or someone flushes the cache, every request misses and the database receives the full load it was never sized for. So never flush everything at peak. Warm caches before they take traffic by preloading the hottest keys. Give each Redis shard a replica so a failover keeps the data. Let new servers ramp up slowly. And cap database concurrency, so misses wait in line instead of crushing it.
- never flush the whole cache
- at peak
- warm before taking traffic:
- preload the top 1M keys
- Redis: replicas, so a failover
- keeps the data
- new app servers: slow start
- (part 2) while local caches
- fill
- DB: a concurrency cap, so
- misses queue, not pile up
What the DB still does
The database still matters. It is the source of truth: two thousand price changes a second, a few hundred to a few thousand reads a second from cache misses, and the checkout reads. That is a modest load: one primary and two replicas for failover and misses. A connection pooler in front caps real connections at about fifty, however many app servers we add. Compare that with the thirty-four replicas we would need without the caches.
writes: 2,000 price changes/s reads: ~250–5,000/s of misses plus checkout reads 1 primary + 2 replicas (failover, misses, checkout) connection pooler: 25 servers × 10 → ~50 real connections vs 34 replicas without caches
The write side
Now the write side. Every request records a view event, so that is a million events a second. Inserting each one into a database is out of the question. Instead, each app server hands events to an asynchronous Kafka producer that groups them into compressed batches of around sixty-four kilobytes every few milliseconds. The request never waits for Kafka’s acknowledgement, and the producer buffer has a fixed size, so a slow Kafka cannot fill the server’s memory.
Size Kafka with the same arithmetic. A million events of a hundred and fifty bytes is a hundred and fifty megabytes a second. Compression shrinks that about four times, to forty. With three copies of every partition, the brokers write a hundred and twenty megabytes a second in total. Forty-eight partitions let many consumers work in parallel. Keeping a day of events is about three and a half terabytes, ten with replication. Six brokers handle it comfortably.
1 view event per request = 1,000,000 events/s one INSERT each: impossible app: async producer, batches of ~64 KB every few ms (linger.ms=5, zstd) never wait for the ack on the hot path; bounded buffer
1,000,000 × 150 B = 150 MB/s compressed ~4x → ~40 MB/s replication factor 3 → ~120 MB/s of disk writes across the brokers 48 partitions for parallel consumers, 6 brokers keep 24 h: 40 MB/s × 86,400 ≈ 3.5 TB × 3 ≈ 10 TB of disk
Batch writers
Batch writers consume the events. They sum views per product per minute in memory, then write large batches, around a hundred thousand rows at a time, into an analytics store such as ClickHouse, which is built for exactly that. They commit Kafka offsets only after a successful write, and use event IDs to ignore duplicates. Raw events are also archived to object storage for replays. Our transactional database never sees a single view event.
- consumers sum views per SKU
- per minute in memory
- write to an analytics store
- (ClickHouse): inserts of
- ~100,000 rows at a time
- commit Kafka offsets after the
- write; dedupe by event id
- raw events also land in object
- storage for replays
- the OLTP DB never sees them
What may be lost
Last, decide what can be lost. View events are analytics. If Kafka is slow and the producer buffer fills, drop them and count how many you dropped. A price read must never fail because analytics is having a bad day. Price changes are different: they are written to the database first and are never lost. Orders do not touch this path at all. Making this explicit is what lets the hot path stay simple and fast.
- view events: analytics, may
- drop under overload, count
- what you dropped
- price changes: never lost,
- written to the DB first
- orders: never on this path
- a price read must not fail
- because Kafka is slow
Say it like this
In an interview: reads go through a two-second CDN micro-cache, an in-process cache, then a six-shard Redis cluster, so the database sees about one read in four thousand. Price changes flow out as events that update every cache layer. I protect against hot keys with local caching and key copies, and against stampedes with single flight and refresh ahead. View events go through Kafka in batches into ClickHouse, never into the main database.
- "Reads: CDN micro-cache for 2 s,
- an in-process cache, then a
- 6-shard Redis cluster, so the
- DB sees ~1 in 4,000 reads.
- Changes flow by events. I guard
- hot keys and stampedes. Views
- go through Kafka in batches
- into ClickHouse."