SYSTEM DESIGN · DEEP DIVE · 1M REQUESTS PER SECOND · PART 3 OF 6

Handling 1 Million Requests per Second, Part 3: Threads, Event Loops, CPU and Memory

Inside one app server at 40,000 requests a second: why thread-per-request caps throughput, how event loops and epoll serve many connections on few threads, Java virtual threads, never blocking the loop, kernel limits (file descriptors, accept backlog, SO_REUSEPORT, conntrack), finding CPU hot spots, serialization and compression costs, garbage collection, contention and timeouts.

Part 3 of 6 in How to Handle 1 Million Requests per Second, a deep dive in 6 videos.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
1 / 22

Press play to hear the narration, or step through at your own pace.

Notes

Inside one server

Follow one request inside the server. Bytes arrive from a proxy connection into the kernel’s socket buffer. The application learns the socket is readable, parses the request, and asks Redis for the price. Then it waits almost a millisecond for the answer, which is an eternity: about four other requests’ worth of CPU. What the server does with its cores during that wait decides how many servers we need.

Thread per request

The classic model is thread per request, the default in servlet containers like Tomcat. Each request owns an operating system thread from start to finish, so code can simply call Redis and block until the answer arrives. It is easy to read and debug. But every thread reserves stack memory and kernel bookkeeping, and switching between thousands of threads costs the CPU a few microseconds each time.

Little’s Law shows the problem. With a pool of two hundred threads and twenty milliseconds per request, the most a server can do is ten thousand requests a second, even if the CPU is mostly idle. We need forty thousand. And if Redis slows to half a second, the same pool manages four hundred a second. You could raise the pool to twenty thousand threads, but then you pay for twenty thousand stacks and a scheduler that spends its time switching.

Thread per request
  • a pool of OS threads; each
  • request owns one until done
  • blocking calls are simple:
  • price = redis.get(key)
  • each thread: ~1 MB stack
  • reserved, kernel bookkeeping
  • waiting threads cost memory;
  • switching costs CPU
  • (a few µs each switch)
Threads cap throughput
max req/s = threads ÷ latency

200 threads ÷ 20 ms
  = 10,000 req/s per server
  we need 40,000

Redis slows to 500 ms:
  200 ÷ 0.5 s = 400 req/s
  queue grows, timeouts, 503s

more threads: 20,000 × 1 MB

Event loops

The alternative is an event loop. One thread per core runs a loop: ask the kernel which sockets are ready, handle each one, and never wait. When a request needs Redis, the loop sends the command, registers a callback, and moves on to the next socket. When the reply arrives, it shows up as another ready event. Sixteen loop threads can serve tens of thousands of connections. This is how Netty, NGINX, Node and Vert.x work.

The kernel interface matters. The old select and poll calls take the full list of sockets on every call, and the kernel checks all of them, so with a hundred thousand connections and a hundred ready you scan a hundred thousand every loop. Linux epoll registers each socket once and keeps a ready list, so each wait costs in proportion to the sockets that actually have work. Big-O applies inside the kernel too.

Event loop
one loop thread per core:

while (true) {
  ready = epoll_wait(...)
  for (s in ready)
    handle(s)  // never blocks
}

Redis call: send the command,
  register a callback, move on
Netty, NGINX, Node.js, Vert.x
Why epoll, not poll
select / poll: pass all sockets
  every call, kernel scans all
  100,000 sockets → O(n) work
  even if only 100 are ready

epoll: register sockets once,
  kernel keeps a ready list
  epoll_wait → O(ready)

Big-O matters inside the kernel
too (kqueue on BSD and macOS)

Never block the loop

Event loops have one golden rule: never block the loop. A blocking database driver, a lock wait, a file read, a DNS lookup, a retry that sleeps, or parsing a huge body. Any of these freezes every connection on that loop. A single fifty millisecond stall delays the hundred or so requests waiting on that loop, and a blocking call on every request drops the loop from two and a half thousand requests a second to twenty. Use non-blocking clients, and send unavoidable blocking work to a small bounded worker pool.

Loop killers
  • blocking JDBC or HTTP client
  • synchronized wait on a lock
  • file reads, DNS lookups
  • parsing a 5 MB JSON body
  • Thread.sleep in a retry
  • one 50 ms stall on 1 of 16
  • loops: ~125 requests late
  • a 50 ms call on every request:
  • that loop does 20 req/s
  • fix: async clients, or a small
  • bounded pool for blocking work

Virtual threads

Java twenty-one offers a middle path: virtual threads. You write ordinary blocking code, but when a virtual thread waits on I/O, the JVM parks it and frees the real carrier thread for other work. Each costs a few kilobytes, so twenty thousand are fine. Go’s goroutines work the same way. Watch for pinning inside synchronized blocks on older JDKs, and remember: they remove the thread limit, not the limit of the database behind you.

Java virtual threads (21+)
  • write simple blocking code
  • the JVM parks the virtual thread
  • on I/O and frees the carrier
  • a few KB each: 20,000 is fine
  • Go goroutines work the same way
  • watch for: pinning inside
  • synchronized (before JDK 24),
  • heavy ThreadLocals
  • they remove the thread limit,
  • not the database limit

Choosing a model

So which model for our price API? The proxies are event loops, always. For the app, either an event-loop framework such as Netty or Spring WebFlux, which is the most efficient but makes code callback-shaped, or virtual threads, which keep code plain and come close in speed. What to avoid is a small fixed pool of operating system threads making blocking calls. Whichever you pick, still limit how many calls can be outstanding to each dependency.

Which model for the price API
  • proxies: event loops (always)
  • app, option A: Netty / WebFlux
  • fastest, callback-style code
  • app, option B: virtual threads
  • plain code, close in speed
  • avoid: fixed pool of 200 OS
  • threads with blocking calls
  • either way: bound concurrency
  • per dependency (semaphores)

Kernel limits

Now the kernel. Every socket is a file descriptor, and many Linux systems still default to a soft limit of one thousand and twenty-four per process. Past it, accept fails with too many open files and connections are refused. Raise the limit for the service, for example to about a million in its systemd unit, and raise the system-wide maximum to match. Then alert at seventy percent, because hitting this limit looks like a mysterious outage.

The next limit is the accept queue. A new connection waits in a kernel queue until the application accepts it. The queue size is the smaller of the backlog your app asks for and the system setting somaxconn. When it fills, the kernel drops connection attempts, and the client retries after about a second, then three. The tell-tale sign is latency spikes of exactly one or three seconds. Raise both settings, and make accepting fast.

A few more. SO REUSE PORT lets each event loop have its own listening socket, so the kernel spreads new connections evenly instead of all loops fighting over one. The network card’s receive queues spread incoming packets across cores. And if a firewall tracks connections, its conntrack table can fill at high connection rates and silently drop packets. Raise its size or skip tracking for this port. Keep every one of these settings in code, not in someone’s memory.

File descriptors
every socket = 1 file descriptor
default soft limit: 1,024
  → "Too many open files"

raise per process:
  LimitNOFILE=1048576 (systemd)
  ulimit -n 1048576
raise system-wide:
  fs.file-max, fs.nr_open

alert at 70% of the limit
The accept backlog
new connections wait in the
  accept queue until accept()
queue size = min(listen backlog,
  net.core.somaxconn)
full → SYNs dropped → client
  retries after ~1 s, then 3 s

symptom: latency spikes of
  exactly 1 s or 3 s
check: nstat ListenOverflows
fix: raise both, accept faster
More kernel and NIC settings
  • SO_REUSEPORT: one listening
  • socket per loop, the kernel
  • spreads new connections
  • NIC receive queues (RSS) spread
  • packets across cores
  • conntrack table full → packets
  • silently dropped: raise
  • nf_conntrack_max or skip
  • tracking for this port
  • record every setting in code

Find the CPU hot spots

To spend less CPU, first find where it goes. Run a sampling profiler such as async-profiler in production, where it costs around one percent, and read the flame graph: the widest bars are your bill. The usual surprises are regular expressions compiled on every request, exceptions used for normal control flow, log lines formatted and then thrown away, reflection-heavy JSON mapping, the body copied several times, and random generators behind a shared lock.

Common CPU surprises
  • regex compiled per request
  • exceptions used for control
  • flow (stack traces are slow)
  • string-formatting log lines
  • that are then discarded
  • reflection-based JSON mapping
  • copying the body 3 times
  • UUID / random from a shared
  • locked generator
  • tool: async-profiler, ~1% cost

Serialization

Serialization is often the biggest single cost. Encoding two kilobytes of JSON and compressing it might cost fifty-five microseconds together, which at a million a second is fifty-five cores. But a price changes two thousand times a second and is read a million times. So encode and compress when the price changes, and cache the finished bytes per product, region and encoding. A request then just copies bytes to the socket. Fifty-five cores saved by moving work from reads to writes.

Format choices follow. The public API stays JSON, because clients expect it, compressed with gzip or brotli for bodies above about a kilobyte. Between internal services, a binary format such as Protobuf is several times smaller and faster to parse. Brotli compresses better but is slower to write, which does not matter if you compress once on change. And code-generated serializers beat reflection-based ones on every request.

Encode once, not per request
JSON encode 2 KB ≈ 30 µs
gzip 2 KB → 600 B ≈ 25 µs
× 1,000,000/s = ~55 cores

a price changes ~2,000×/s
  but is read 1,000,000×/s
so: encode + gzip on change,
  cache the final bytes
  per (sku, region, encoding)
request: copy bytes to socket
→ ~55 cores saved
Formats and compression
  • public API: JSON, gzip or
  • brotli for large bodies
  • service to service: Protobuf
  • is 3–10x smaller, faster
  • skip compression under ~1 KB
  • brotli: smaller, slower to
  • write; fine if done once
  • zstd: fast, good ratio
  • generated codecs beat
  • reflection

Memory and GC

Now memory. In part one we saw how garbage per request becomes collections per second. The collector decides how long each one stops the world. G1 aims for pauses of tens of milliseconds. Generational ZGC, in Java twenty-one and later, keeps pauses under a millisecond at the cost of more CPU and memory. Large in-process caches add live objects to scan, so bound them by size. Fix the heap size, and leave memory for the kernel’s buffers.

The best collector setting is allocating less. Use pooled buffers instead of fresh byte arrays. Write responses straight into the output buffer rather than building a big string. Avoid boxing numbers. Reuse parsers and codecs. Going from twenty kilobytes to three per request drops the garbage rate from eight hundred to a hundred and twenty megabytes a second per server, and a two gigabyte young generation fills every seventeen seconds instead of every two and a half.

Garbage collection
  • allocation rate × time =
  • young collections per minute
  • G1: pause goal ~10–50 ms
  • generational ZGC (JDK 21+):
  • pauses under 1 ms, costs
  • more CPU and memory
  • big on-heap caches = more to
  • scan: bound them by size
  • heap: fixed -Xms = -Xmx,
  • leave RAM for the OS
Allocate less
before: ~20 KB per request
  → 800 MB/s per server
pooled direct buffers (Netty)
write JSON straight to the
  output buffer, no big String
no boxing: long, not Long
reuse parsers, codecs, arrays
after: ~3 KB per request
  → 120 MB/s per server
  → young GC every ~17 s,
    not every 2.5 s

Contention

With sixteen cores, shared state becomes the bottleneck. A single atomic counter for a metric makes every core fight over the same cache line. Use a striped counter like LongAdder. A synchronized LRU map serialises every lookup behind one lock. Use a concurrent cache such as Caffeine. A shared random generator becomes a queue. The rule: keep state per core or per thread, and merge it when you read, not when you write.

Shared state between cores
  • one AtomicLong for a metric:
  • 16 cores bounce one cache
  • line → use LongAdder
  • synchronized LRU map: global
  • lock → use Caffeine
  • shared Random → use
  • ThreadLocalRandom
  • rule: per-core state, merge
  • when you read

Timeouts and limits

Finally, bound everything. Every outbound call has a timeout: twenty milliseconds for Redis, not the library default of a minute. Little’s Law sizes the connection pools: forty thousand requests a second times one millisecond is about forty Redis calls in flight per server, so a few pipelined connections with a cap of a few hundred is ample. The database gets a tiny concurrency limit. And beyond four thousand requests in flight, the server answers a fast five oh three instead of queuing forever.

Bound everything
Redis timeout: 20 ms
Redis in flight per server:
  40,000 req/s × 1 ms = 40
  → pipelined connections, cap
    in-flight at ~500
DB: max 10 concurrent per server
max in-flight requests per server:
  4,000, beyond that → fast 503
request deadline passed along:
  no work after the client left

One server

Here is one server, summarised. Sixteen cores serving forty thousand requests a second, about half of them busy. Sixteen event loops with their own listening sockets. Around a thousand pooled connections from the proxies. File descriptor and backlog limits raised. Responses pre-encoded in the cache. An eight gigabyte heap with a low-pause collector and about three kilobytes of garbage per request. A ninety-ninth percentile under five milliseconds, and a hard cap on requests in flight.

One server, summarised
16 cores, 40,000 req/s
~8 cores busy at 200 µs each
16 event loops, SO_REUSEPORT
~1,000 pooled conns from proxies
nofile 1,048,576, somaxconn 4096
pre-encoded responses in cache
heap 8 GB, young 2 GB, ZGC
~3 KB garbage per request
p99 in the server: under 5 ms
max 4,000 in flight, then 503

Say it like this

In an interview: each server takes forty thousand requests a second. A pool of two hundred blocking threads caps out at ten thousand at twenty milliseconds, so I use event loops or virtual threads. I raise file descriptor and backlog limits. I cache pre-encoded responses so most requests do no serialization, cut allocation to keep collections rare, and bound every outbound call with a timeout and a concurrency limit.

In the interview
  • "Each server takes 40k req/s.
  • A pool of 200 blocking threads
  • caps at 10k at 20 ms, so I use
  • event loops or virtual threads.
  • I raise file and backlog limits,
  • cache pre-encoded responses,
  • cut allocation, and bound every
  • call with timeouts and limits."

Interview questions

Why does thread-per-request struggle at high request rates?

Throughput is capped at threads divided by latency, so 200 threads at 20 ms give only 10,000 requests per second, and far less when a dependency slows. Adding thousands of OS threads costs memory and context switching.

How does an event loop handle many connections with few threads?

Each loop thread asks the kernel (epoll) which sockets are ready, handles them without blocking, and registers callbacks for outbound I/O. One thread per core can serve tens of thousands of connections because no thread waits idle.

Why is epoll better than select or poll?

select and poll pass and scan every socket on each call, which is O(n) in connections. epoll registers sockets once and returns only the ready ones, so the cost follows activity rather than the number of open connections.

What happens if you block an event loop?

Every connection on that loop stalls until the blocking call returns. Use non-blocking clients and move unavoidable blocking work, such as a legacy JDBC driver, to a small bounded worker pool.

What do Java virtual threads change?

They let you write blocking-style code while the JVM parks waiting virtual threads and reuses a few carrier threads, so tens of thousands of concurrent requests are cheap. They do not remove limits of downstream systems, so concurrency per dependency must still be bounded.

Which Linux limits matter for a high-traffic server?

Open file descriptors per process (each socket is one), the accept queue (listen backlog and net.core.somaxconn), the ephemeral port range for outgoing connections, and the conntrack table if connection tracking is on.

How can you cut serialization cost for a read-heavy API?

Encode and compress a response when the underlying data changes and cache the final bytes, so each read only copies bytes to the socket. Use generated codecs and binary formats such as Protobuf between internal services.

How do you reduce garbage collection impact?

Allocate less per request with pooled buffers, streaming writers and no boxing, bound in-process caches, and choose a low-pause collector such as generational ZGC when tail latency matters more than a little extra CPU.