Notes
How big systems fail
Here is the list of amplifiers to design against. Retries multiply load exactly when the system is weakest. Herds of clients arrive at the same instant. Hot spots concentrate load on one shard. Slow nodes hold on to requests. Slowness grows the amount of work in flight, as Little’s Law told us. And most incidents start with a change: a deploy or a config push. Each amplifier needs a damper, chosen calmly before the incident, not during it.
- retries multiply load
- herds arrive all at once
- hot spots concentrate load
- slow nodes hold requests
- slowness grows in-flight work
- (Little’s Law)
- changes: deploys and config
- each needs a damper, chosen
- before the incident
Retry storms
Retry storms first. If the client, the proxy and the app server each try once and retry three times, one failing request at the bottom can be attempted four times four times four: sixty-four times. Suppose Redis blips for two seconds and ten percent of calls fail. With retries at every layer, its normal fifty thousand operations a second can become several hundred thousand, far above its capacity, so the two-second blip becomes a real outage.
The dampers. Retry at one layer only, usually the one nearest the user. Give retries a budget: no more than about ten percent extra requests, and stop retrying once the budget is spent. Back off exponentially with full jitter, meaning a random wait up to a growing cap, so retries do not arrive in waves. Retry only idempotent calls. Never retry an overload response. And pass a deadline down with each request, so no layer retries work that the caller has already abandoned.
each layer: 1 try + 3 retries = 4 attempts client → L7 → app → Redis 4 × 4 × 4 = 64 attempts for one user request Redis blips for 2 s, 10% fail normal 50,000 ops/s → several hundred thousand ops/s of retries → the blip becomes an outage
- retry at one layer only,
- usually nearest the user
- retry budget: retries ≤ 10%
- of requests, else stop
- exponential backoff + full
- jitter: sleep random(0, cap)
- retry only idempotent calls
- never retry 503 overload or
- 429; honour Retry-After
- propagate deadlines: no retry
- after the caller gave up
Thundering herd
Next, the thundering herd. After a thirty-second outage, two million clients reconnect in the same second. Each full TLS handshake costs about a tenth of a millisecond, so that is two hundred core-seconds of work: every proxy core in all three regions flat out for more than a second, with real requests waiting behind it. Herds also come from apps that refresh on the hour, push campaigns that open millions of apps at once, and cache entries all set with the same expiry.
To tame herds, spread the return over time and make returning cheap. Client libraries reconnect after a random delay, spread over thirty to sixty seconds. TLS session tickets let returning clients skip the costly handshake. The edge admits new connections at a capped rate while serving existing ones. Add random jitter to every timer and cache expiry, stagger push notification campaigns, and warm the caches before you let traffic back in.
30 s outage, then recovery 2,000,000 clients reconnect in the same second full TLS handshake ≈ 0.1 ms → 200 core-seconds of work → all ~190 proxy cores at 100% for over 1 s, real traffic waits behind it also: timers at :00, push campaigns, TTLs set together
- clients: reconnect with random
- delay, spread over 30–60 s
- session tickets: resumptions
- skip the costly handshake
- edge: admission control,
- accept new connections at
- a capped rate
- jitter every timer and TTL
- stagger push campaigns
- warm caches before reopening
Hot spots
Hot spots go beyond the hot cache key from part four. If view events are keyed by product, a flash deal sends a third of them into one Kafka partition and its consumer falls behind. Views do not need ordering, so spread them randomly. A single partner sending thirty percent of all traffic needs its own rate limit. The general habit: for every sharded thing, graph the busiest shard against the median shard, and alert when they drift apart.
- Kafka keyed by SKU: a flash
- deal fills one partition;
- key views randomly instead,
- order is not needed
- one tenant or API key at 30%
- of traffic: per-key limits
- one region’s prices popular
- worldwide: replicate them
- look for skew: max vs median
- per shard, per partition
Slow is worse than dead
A slow server is worse than a dead one. A dead server fails health checks and is removed in seconds. A server stuck in a three hundred millisecond garbage collection pause, or with a failing disk, keeps accepting work: at forty thousand requests a second, twelve thousand requests pile up behind one pause. So proxies should eject servers whose latency is an outlier, not only those returning errors. For reads, hedging helps: if no reply arrives by the ninety-fifth percentile, send a second copy elsewhere.
one server pauses 300 ms × 40,000 req/s = 12,000 requests stuck behind it a dead server is ejected in seconds; a slow one is not outlier ejection on latency, not only on errors hedged reads: if no reply by p95, ask a second server (≈5% extra load)
Cascading failure
Put those together and you get a cascading failure. A dependency slows from twenty to five hundred milliseconds. Requests in flight per server jump from eight hundred to twenty thousand. Memory rises, garbage collection pauses get longer, health checks time out, and servers are ejected. The survivors receive even more traffic, so they slow down too, while clients retry. Within a minute, a slow dependency has become a total outage. The cure is to refuse work early.
- dependency slows 20 → 500 ms
- in flight per server 800 →
- 20,000 (Little’s Law)
- memory up → longer GC pauses
- health checks time out →
- servers ejected
- survivors get more traffic →
- they slow down too
- clients retry → more load
- result: total outage
Load shedding
Load shedding is that cure. When requests in flight pass a limit, or time spent queued passes twenty milliseconds, the server answers immediately with a five oh three. A fast refusal costs a few microseconds; a slow success holds memory and ends in a timeout anyway. Shed by priority: analytics and prefetches first, product tiles next, the cart and checkout last. Adaptive concurrency limits, which raise and lower the cap based on measured latency much like TCP does, can tune this automatically.
reject when: in flight > 4,000 or queue wait > 20 ms a fast 503 costs ~5 µs; a slow success costs memory, a thread and a timeout shed by priority: 1 analytics, prefetch 2 product tiles 3 cart and checkout: last adaptive limits (like TCP)
Breakers and bulkheads
Two more dampers. A circuit breaker per dependency watches errors: if half the calls fail over ten seconds, it opens, and calls fail fast to a fallback instead of waiting. For prices, the fallback is the last known value from the local cache, marked as possibly stale. After a pause the breaker lets a few probes through and closes if they succeed. Bulkheads give each dependency its own pool and limit, so a slow delivery-estimate service cannot use up capacity the price lookup needs.
- breaker per dependency: 50%
- errors over 10 s → open:
- fail fast, use a fallback
- → half-open: a few probes
- fallback for prices: last known
- value from the local cache,
- flagged as possibly stale
- bulkheads: separate pools and
- limits per dependency, so
- delivery estimates cannot
- eat the price lane
Graceful degradation
Plan graceful degradation as an ordered list. Step one, drop analytics view events and count them. Step two, skip the expensive delivery estimate. Step three, serve prices up to a minute stale from local caches if Redis is in trouble. Step four, let the CDN serve stale responses when our origin errors. Throughout, checkout keeps verifying the price against the database, so stale data can never cost the business money. Deciding this order in advance turns a crisis into a checklist.
- 1 drop view events (counted)
- 2 skip delivery estimates
- 3 serve prices up to 60 s
- stale from local cache
- 4 CDN serves stale on error:
- stale-if-error=300
- 5 checkout still verifies
- price in the database
- decide the order in advance
Changes cause outages
Most outages start with a change, so changes need the same care as traffic. Every deploy starts as a canary on one server, compared for ten minutes against the rest on latency, errors and CPU per request. Then roll three servers at a time. With twenty-two of twenty-five serving, each takes forty-five thousand requests a second at about fifty-six percent busy, which our headroom covers. Config pushes are canaried too, rollbacks are automatic, and nothing big ships at peak.
canary: 1 server, ~4% of a region, for 10 min compare p99, errors, CPU µs per request vs the rest then roll 3 servers at a time 22 carry 1M: 45,000 each, ~56% busy ✓ config pushes: canaried too auto rollback on SLO burn no big changes at peak
Losing a region
Losing a region tests everything at once. With three regions each sized for half a million, capacity is fine. The trap is caches. The surviving regions’ caches are full of their own users’ keys, so the moved traffic misses, and database load spikes exactly when you can least afford it. The fix: the price updater from part four writes every region’s keys into every region’s Redis, so failover lands on warm caches. And practise it with planned game days, not for the first time during a real outage.
- 3 regions, each sized for 500k
- one fails: DNS and anycast
- move its users in minutes
- trap: the survivors’ caches
- hold their own regions’
- keys, misses spike
- fix: the price updater writes
- every region’s keys into
- every region’s Redis
- practice it: game days
The bill
Now the bill, with assumed prices to replace with your own. Six hundred app cores in three regions at twenty-nine dollars each: about seventeen thousand a month. Proxies five and a half thousand, Redis nine, Kafka and ClickHouse eight, databases three. Bandwidth: compressed responses at the average rate are about four hundred and sixty-seven terabytes a month, around twenty-three thousand dollars. Total: about sixty-eight thousand. Unsampled logs, at fifty cents a gigabyte, would add a hundred and fifty thousand.
Turn the bill into one number: cost per million requests. At an average of three hundred thousand a second, we serve about seven hundred and seventy-eight billion requests a month. Sixty-eight thousand dollars divided by that is roughly nine cents per million requests. Track it next to latency, per release. If it rises, something got slower, bigger or chattier, and you can find it before the invoice does.
app: 600 cores × $29 $17,400
L7 proxies: 192 cores $5,600
Redis: 36 nodes × $250 $9,000
Kafka + ClickHouse $8,000
database, 3 regions $3,000
egress: 467 TB × $0.05 $23,300
logs, sampled $2,000
--------
total ≈ $68,000
logs unsampled at $0.50/GB:
+$154,000 average 300,000 req/s ≈ 777.6 billion req/month $68,000 ÷ 777,600 million ≈ $0.09 per million requests track it like latency: per build, per release if it rises, something got slower or bigger
Where to save
Where do the savings come from? Bytes first: compress, remove fields clients do not use, and cache at the edge. Then logs: sample, but keep every error. Then CPU per request, with the tricks from part three. Scale the stateless fleet with the daily curve, since the average is only thirty percent of the peak, and run it on cheaper machine types or committed pricing. But do not save on headroom, regions or replicas. Those are what you are really paying for.
- bytes: compress, trim fields,
- cache at the edge
- logs: sample, keep errors
- CPU: encode once, cut allocs
- scale with the daily curve:
- average is 30% of the peak
- cheaper machines for stateless
- servers: ARM, spot, commits
- do not save on: headroom,
- regions, replicas
The final architecture
Here is the final architecture. Users reach an anycast CDN that finishes TLS and micro-caches prices for two seconds, answering half the traffic. The rest flows through layer-four and layer-seven balancers to event-loop app servers, which answer ninety percent from their local cache and most of the remainder from Redis. The database is the source of truth, feeding price changes back out as events. View events go through Kafka into ClickHouse. Telemetry watches all of it, in three active regions.
And the numbers, per region, sized for half a million requests a second. About half answered by the CDN. Four layer-four boxes and eight eight-core proxies. Thirteen sixteen-core app servers plus spares, at two hundred microseconds of CPU per request and no more than half busy. Local caches at ninety percent, Redis at ninety-nine and a half. Six Redis shards with replicas, a small database, six Kafka brokers. A ninety-ninth percentile under twenty-five milliseconds, at about nine cents per million requests.
sized for 500,000 req/s CDN micro-cache: ~50% hit L4: 4 boxes · L7: 8 × 8 cores app: ~13 × 16 cores (+ spares) 200 µs CPU, ≤50% busy local cache 90% · Redis 99.5% Redis: 6 shards + 6 replicas DB: primary + 2 replicas Kafka: 6 brokers, 48 parts p99 under 25 ms · 9¢ per 1M
Say it like this
In an interview, close with risks and cost. The biggest risks are amplifiers, so I retry at one layer only, with a budget and jitter. I shed load early by priority, use circuit breakers with stale fallbacks, and canary every deploy and config change. Three active regions are each sized for half a million requests a second, with caches pre-warmed for failover. And it costs about nine cents per million requests, which I track per release.
- "Biggest risks are amplifiers:
- I retry at one layer with a
- budget and jitter, shed load
- early by priority, use breakers
- with stale fallbacks, canary
- every change, and size 3
- regions for 500k each. It costs
- ~9¢ per million requests."