Notes
Inside one server
Follow one request inside the server. Bytes arrive from a proxy connection into the kernel’s socket buffer. The application learns the socket is readable, parses the request, and asks Redis for the price. Then it waits almost a millisecond for the answer, which is an eternity: about four other requests’ worth of CPU. What the server does with its cores during that wait decides how many servers we need.
Thread per request
The classic model is thread per request, the default in servlet containers like Tomcat. Each request owns an operating system thread from start to finish, so code can simply call Redis and block until the answer arrives. It is easy to read and debug. But every thread reserves stack memory and kernel bookkeeping, and switching between thousands of threads costs the CPU a few microseconds each time.
Little’s Law shows the problem. With a pool of two hundred threads and twenty milliseconds per request, the most a server can do is ten thousand requests a second, even if the CPU is mostly idle. We need forty thousand. And if Redis slows to half a second, the same pool manages four hundred a second. You could raise the pool to twenty thousand threads, but then you pay for twenty thousand stacks and a scheduler that spends its time switching.
- a pool of OS threads; each
- request owns one until done
- blocking calls are simple:
- price = redis.get(key)
- each thread: ~1 MB stack
- reserved, kernel bookkeeping
- waiting threads cost memory;
- switching costs CPU
- (a few µs each switch)
max req/s = threads ÷ latency 200 threads ÷ 20 ms = 10,000 req/s per server we need 40,000 Redis slows to 500 ms: 200 ÷ 0.5 s = 400 req/s queue grows, timeouts, 503s more threads: 20,000 × 1 MB
Event loops
The alternative is an event loop. One thread per core runs a loop: ask the kernel which sockets are ready, handle each one, and never wait. When a request needs Redis, the loop sends the command, registers a callback, and moves on to the next socket. When the reply arrives, it shows up as another ready event. Sixteen loop threads can serve tens of thousands of connections. This is how Netty, NGINX, Node and Vert.x work.
The kernel interface matters. The old select and poll calls take the full list of sockets on every call, and the kernel checks all of them, so with a hundred thousand connections and a hundred ready you scan a hundred thousand every loop. Linux epoll registers each socket once and keeps a ready list, so each wait costs in proportion to the sockets that actually have work. Big-O applies inside the kernel too.
one loop thread per core:
while (true) {
ready = epoll_wait(...)
for (s in ready)
handle(s) // never blocks
}
Redis call: send the command,
register a callback, move on
Netty, NGINX, Node.js, Vert.x select / poll: pass all sockets every call, kernel scans all 100,000 sockets → O(n) work even if only 100 are ready epoll: register sockets once, kernel keeps a ready list epoll_wait → O(ready) Big-O matters inside the kernel too (kqueue on BSD and macOS)
Never block the loop
Event loops have one golden rule: never block the loop. A blocking database driver, a lock wait, a file read, a DNS lookup, a retry that sleeps, or parsing a huge body. Any of these freezes every connection on that loop. A single fifty millisecond stall delays the hundred or so requests waiting on that loop, and a blocking call on every request drops the loop from two and a half thousand requests a second to twenty. Use non-blocking clients, and send unavoidable blocking work to a small bounded worker pool.
- blocking JDBC or HTTP client
- synchronized wait on a lock
- file reads, DNS lookups
- parsing a 5 MB JSON body
- Thread.sleep in a retry
- one 50 ms stall on 1 of 16
- loops: ~125 requests late
- a 50 ms call on every request:
- that loop does 20 req/s
- fix: async clients, or a small
- bounded pool for blocking work
Virtual threads
Java twenty-one offers a middle path: virtual threads. You write ordinary blocking code, but when a virtual thread waits on I/O, the JVM parks it and frees the real carrier thread for other work. Each costs a few kilobytes, so twenty thousand are fine. Go’s goroutines work the same way. Watch for pinning inside synchronized blocks on older JDKs, and remember: they remove the thread limit, not the limit of the database behind you.
- write simple blocking code
- the JVM parks the virtual thread
- on I/O and frees the carrier
- a few KB each: 20,000 is fine
- Go goroutines work the same way
- watch for: pinning inside
- synchronized (before JDK 24),
- heavy ThreadLocals
- they remove the thread limit,
- not the database limit
Choosing a model
So which model for our price API? The proxies are event loops, always. For the app, either an event-loop framework such as Netty or Spring WebFlux, which is the most efficient but makes code callback-shaped, or virtual threads, which keep code plain and come close in speed. What to avoid is a small fixed pool of operating system threads making blocking calls. Whichever you pick, still limit how many calls can be outstanding to each dependency.
- proxies: event loops (always)
- app, option A: Netty / WebFlux
- fastest, callback-style code
- app, option B: virtual threads
- plain code, close in speed
- avoid: fixed pool of 200 OS
- threads with blocking calls
- either way: bound concurrency
- per dependency (semaphores)
Kernel limits
Now the kernel. Every socket is a file descriptor, and many Linux systems still default to a soft limit of one thousand and twenty-four per process. Past it, accept fails with too many open files and connections are refused. Raise the limit for the service, for example to about a million in its systemd unit, and raise the system-wide maximum to match. Then alert at seventy percent, because hitting this limit looks like a mysterious outage.
The next limit is the accept queue. A new connection waits in a kernel queue until the application accepts it. The queue size is the smaller of the backlog your app asks for and the system setting somaxconn. When it fills, the kernel drops connection attempts, and the client retries after about a second, then three. The tell-tale sign is latency spikes of exactly one or three seconds. Raise both settings, and make accepting fast.
A few more. SO REUSE PORT lets each event loop have its own listening socket, so the kernel spreads new connections evenly instead of all loops fighting over one. The network card’s receive queues spread incoming packets across cores. And if a firewall tracks connections, its conntrack table can fill at high connection rates and silently drop packets. Raise its size or skip tracking for this port. Keep every one of these settings in code, not in someone’s memory.
every socket = 1 file descriptor default soft limit: 1,024 → "Too many open files" raise per process: LimitNOFILE=1048576 (systemd) ulimit -n 1048576 raise system-wide: fs.file-max, fs.nr_open alert at 70% of the limit
new connections wait in the accept queue until accept() queue size = min(listen backlog, net.core.somaxconn) full → SYNs dropped → client retries after ~1 s, then 3 s symptom: latency spikes of exactly 1 s or 3 s check: nstat ListenOverflows fix: raise both, accept faster
- SO_REUSEPORT: one listening
- socket per loop, the kernel
- spreads new connections
- NIC receive queues (RSS) spread
- packets across cores
- conntrack table full → packets
- silently dropped: raise
- nf_conntrack_max or skip
- tracking for this port
- record every setting in code
Find the CPU hot spots
To spend less CPU, first find where it goes. Run a sampling profiler such as async-profiler in production, where it costs around one percent, and read the flame graph: the widest bars are your bill. The usual surprises are regular expressions compiled on every request, exceptions used for normal control flow, log lines formatted and then thrown away, reflection-heavy JSON mapping, the body copied several times, and random generators behind a shared lock.
- regex compiled per request
- exceptions used for control
- flow (stack traces are slow)
- string-formatting log lines
- that are then discarded
- reflection-based JSON mapping
- copying the body 3 times
- UUID / random from a shared
- locked generator
- tool: async-profiler, ~1% cost
Serialization
Serialization is often the biggest single cost. Encoding two kilobytes of JSON and compressing it might cost fifty-five microseconds together, which at a million a second is fifty-five cores. But a price changes two thousand times a second and is read a million times. So encode and compress when the price changes, and cache the finished bytes per product, region and encoding. A request then just copies bytes to the socket. Fifty-five cores saved by moving work from reads to writes.
Format choices follow. The public API stays JSON, because clients expect it, compressed with gzip or brotli for bodies above about a kilobyte. Between internal services, a binary format such as Protobuf is several times smaller and faster to parse. Brotli compresses better but is slower to write, which does not matter if you compress once on change. And code-generated serializers beat reflection-based ones on every request.
JSON encode 2 KB ≈ 30 µs gzip 2 KB → 600 B ≈ 25 µs × 1,000,000/s = ~55 cores a price changes ~2,000×/s but is read 1,000,000×/s so: encode + gzip on change, cache the final bytes per (sku, region, encoding) request: copy bytes to socket → ~55 cores saved
- public API: JSON, gzip or
- brotli for large bodies
- service to service: Protobuf
- is 3–10x smaller, faster
- skip compression under ~1 KB
- brotli: smaller, slower to
- write; fine if done once
- zstd: fast, good ratio
- generated codecs beat
- reflection
Memory and GC
Now memory. In part one we saw how garbage per request becomes collections per second. The collector decides how long each one stops the world. G1 aims for pauses of tens of milliseconds. Generational ZGC, in Java twenty-one and later, keeps pauses under a millisecond at the cost of more CPU and memory. Large in-process caches add live objects to scan, so bound them by size. Fix the heap size, and leave memory for the kernel’s buffers.
The best collector setting is allocating less. Use pooled buffers instead of fresh byte arrays. Write responses straight into the output buffer rather than building a big string. Avoid boxing numbers. Reuse parsers and codecs. Going from twenty kilobytes to three per request drops the garbage rate from eight hundred to a hundred and twenty megabytes a second per server, and a two gigabyte young generation fills every seventeen seconds instead of every two and a half.
- allocation rate × time =
- young collections per minute
- G1: pause goal ~10–50 ms
- generational ZGC (JDK 21+):
- pauses under 1 ms, costs
- more CPU and memory
- big on-heap caches = more to
- scan: bound them by size
- heap: fixed -Xms = -Xmx,
- leave RAM for the OS
before: ~20 KB per request
→ 800 MB/s per server
pooled direct buffers (Netty)
write JSON straight to the
output buffer, no big String
no boxing: long, not Long
reuse parsers, codecs, arrays
after: ~3 KB per request
→ 120 MB/s per server
→ young GC every ~17 s,
not every 2.5 s Contention
With sixteen cores, shared state becomes the bottleneck. A single atomic counter for a metric makes every core fight over the same cache line. Use a striped counter like LongAdder. A synchronized LRU map serialises every lookup behind one lock. Use a concurrent cache such as Caffeine. A shared random generator becomes a queue. The rule: keep state per core or per thread, and merge it when you read, not when you write.
- one AtomicLong for a metric:
- 16 cores bounce one cache
- line → use LongAdder
- synchronized LRU map: global
- lock → use Caffeine
- shared Random → use
- ThreadLocalRandom
- rule: per-core state, merge
- when you read
Timeouts and limits
Finally, bound everything. Every outbound call has a timeout: twenty milliseconds for Redis, not the library default of a minute. Little’s Law sizes the connection pools: forty thousand requests a second times one millisecond is about forty Redis calls in flight per server, so a few pipelined connections with a cap of a few hundred is ample. The database gets a tiny concurrency limit. And beyond four thousand requests in flight, the server answers a fast five oh three instead of queuing forever.
Redis timeout: 20 ms
Redis in flight per server:
40,000 req/s × 1 ms = 40
→ pipelined connections, cap
in-flight at ~500
DB: max 10 concurrent per server
max in-flight requests per server:
4,000, beyond that → fast 503
request deadline passed along:
no work after the client left One server
Here is one server, summarised. Sixteen cores serving forty thousand requests a second, about half of them busy. Sixteen event loops with their own listening sockets. Around a thousand pooled connections from the proxies. File descriptor and backlog limits raised. Responses pre-encoded in the cache. An eight gigabyte heap with a low-pause collector and about three kilobytes of garbage per request. A ninety-ninth percentile under five milliseconds, and a hard cap on requests in flight.
16 cores, 40,000 req/s ~8 cores busy at 200 µs each 16 event loops, SO_REUSEPORT ~1,000 pooled conns from proxies nofile 1,048,576, somaxconn 4096 pre-encoded responses in cache heap 8 GB, young 2 GB, ZGC ~3 KB garbage per request p99 in the server: under 5 ms max 4,000 in flight, then 503
Say it like this
In an interview: each server takes forty thousand requests a second. A pool of two hundred blocking threads caps out at ten thousand at twenty milliseconds, so I use event loops or virtual threads. I raise file descriptor and backlog limits. I cache pre-encoded responses so most requests do no serialization, cut allocation to keep collections rare, and bound every outbound call with a timeout and a concurrency limit.
- "Each server takes 40k req/s.
- A pool of 200 blocking threads
- caps at 10k at 20 ms, so I use
- event loops or virtual threads.
- I raise file and backlog limits,
- cache pre-encoded responses,
- cut allocation, and bound every
- call with timeouts and limits."