SYSTEM DESIGN · TRACK 7
Deep Dives
Long, multi-part series that follow one hard question all the way down: the math, the network, the code, the caches, the measurements, the failures and the bill. Each part is its own video.
All 6 are live. Each one is an animated walkthrough you can step through, written notes, and the interview questions with answers.
How to Handle 1 Million Requests per Second
What really changes when one service must answer a million HTTP requests every second: budgets, connections, caches, measurement, failures and cost.
Series overview: all 6 parts →
- 1 The scale and the per-request budgetWhat a million requests a second really means: one request every microsecond, the CPU budget per request, the rule that one microsecond per request costs one core, how Big-O and memory allocation turn into servers and money, Little’s Law for requests in flight, why one-in-a-million events happen every second, and the bandwidth and logging bill before any code is written.
- 2 The network edge and load balancingHow a million requests a second reach the app servers: what a new TCP and TLS connection costs, keep-alive, HTTP/2 and HTTP/3, the load balancing tiers (GeoDNS, anycast, CDN, L4, L7) and the capacity of each, balancing algorithms, the long-lived connection trap, ephemeral port exhaustion, health checks and draining, and multiple regions.
- 3 The app tier: threads, event loops, CPU and memoryInside one app server at 40,000 requests a second: why thread-per-request caps throughput, how event loops and epoll serve many connections on few threads, Java virtual threads, never blocking the loop, kernel limits (file descriptors, accept backlog, SO_REUSEPORT, conntrack), finding CPU hot spots, serialization and compression costs, garbage collection, contention and timeouts.
- 4 Caching and keeping the database out of the hot pathWhy the database must not serve a million reads a second, and how layered caches make sure it never has to: CDN micro-caching, an in-process cache, a Redis cluster, hit-rate math, hot keys, stampedes, invalidation through change events, cold starts, then the write side: batching a million view events a second through Kafka into an analytics store.
- 5 Load testing and measuring latency properlyHow to prove a service can take a million requests a second: generating the load from a fleet of machines, open versus closed load models, coordinated omission and how it hides slow requests, percentiles and why you never average them, realistic traffic, test shapes, finding the knee, hunting bottlenecks, testing the test, and production observability, SLOs and performance checks in CI.
- 6 Failure modes, cost and the final architectureHow a million-request-a-second service fails and how to stop it: retry storms, thundering herds, hot spots, GC pauses and slow nodes, cascading failure, load shedding, circuit breakers, graceful degradation, deploys and region loss. Then the monthly bill line by line, cost per million requests, where to save, and the final architecture with every number from the series.
The whole path
Concepts, technologies, building blocks, patterns, the LADDER method and the design problems.