Notes
The problem
Two naive designs come up a lot. The first starts a new thread for every order. At lunch that means hundreds of threads, each with its own stack, fighting for a few CPU cores, until the process runs out of memory. The second shares a plain list and spins in a loop waiting for work, which burns a whole core doing nothing, and the list is not even thread safe.
// 1: a thread per order
new Thread(() -> cook(order)).start();
// 500 orders = 500 threads
// 2: a shared list + busy loop
while (orders.isEmpty()) { } // burns CPU
cook(orders.remove(0)); // not safe Producer-consumer
The producer consumer pattern puts a queue between the side that creates work and the side that does it. The order intake is the producer: it puts orders into the queue. Cooks are consumers: they take orders out. Neither side knows about the other. You can add cooks at lunch without touching intake code, and a slow cook does not slow down taking orders, up to the queue's limit.
BlockingQueue
Java gives you this queue ready made: BlockingQueue. Take puts the cook to sleep while the queue is empty and wakes it the moment an order arrives, so no CPU is wasted. Put does the opposite: if the queue is full, the producer sleeps until there is room. The queue handles all the locking inside. For a deadline, offer and poll accept a timeout and return false or null instead of waiting forever.
BlockingQueue<Order> q =
new ArrayBlockingQueue<>(4);
// producer
q.put(order); // waits if full
// consumer
Order o = q.take(); // waits if empty
offer(o, 50, MILLISECONDS) // or give up Back-pressure
Always give the queue a limit. An unbounded queue looks friendly, but when producers are faster than consumers, it grows until memory runs out, and every order waits longer and longer. A bounded queue gives back pressure. When it is full, the producer must wait, fail fast, or tell the customer the kitchen is busy. A clear no in one second beats a silent forty minute wait.
Thread pools
A thread pool is producer consumer packaged for you. An ExecutorService keeps a fixed set of worker threads and an internal queue of tasks. You submit a task and get back a Future, which you can use later to read the result or the exception. Threads are created once and reused, so a burst of a thousand tasks still uses only the threads you allowed.
ExecutorService pool =
Executors.newFixedThreadPool(2);
Future<Integer> f =
pool.submit(() -> cook(order));
int minutes = f.get();
// a pool IS a queue + workers Sizing a pool
How many threads? For work that keeps the CPU busy, like resizing images, about one thread per core; more just adds switching. For work that mostly waits, like calling a payment service, you can use more, roughly cores times one plus the ratio of wait time to work time. Then measure and tune. Keep slow work in its own pool, so a stuck payment provider cannot starve the fast menu lookups.
- CPU-bound (resize images):
- threads about = cores
- IO-bound (call payment API):
- threads = cores x
- (1 + wait time / work time)
- measure, then tune
- separate pools for slow and
- fast work (bulkheads)
Rejection policies
One trap: the factory method for a fixed thread pool uses an unbounded queue. For production, build a ThreadPoolExecutor yourself with a bounded queue and a rejection policy. The default policy throws an exception. Caller runs policy makes the submitting thread do the task itself, which naturally slows the producer, a neat form of back pressure. The discard policies drop work, which is only fine for things like metrics.
new ThreadPoolExecutor(
2, 2, 0, SECONDS,
new ArrayBlockingQueue<>(4),
new CallerRunsPolicy());
AbortPolicy: throw (default)
CallerRunsPolicy: caller runs it,
which slows the producer
Discard / DiscardOldest: drop Stopping consumers
How does a consumer know the day is over? A common trick is the poison pill: a special marker object put into the queue after the last real order. A cook that takes the marker stops its loop and exits. Put one marker per consumer, so every cook gets one. Because the queue is first in first out, all real orders ahead of the markers are cooked first. Nothing is lost.
static final Order STOP =
new Order(-1, "stop");
while (true) {
Order o = q.take();
if (o == STOP) return;
cook(o);
}
// producer: one STOP per consumer Graceful shutdown
For a pool, graceful shutdown is three steps. Call shutdown: no new tasks are accepted, but queued and running ones continue. Then await termination with a time limit. If the pool is still busy after that, call shutdown now, which interrupts the workers and returns the tasks that never started. Workers should respond to interruption by tidying up and exiting, not by swallowing the exception.
pool.shutdown(); // no new tasks
if (!pool.awaitTermination(
5, TimeUnit.SECONDS)) {
pool.shutdownNow(); // interrupt
}
// workers: handle interrupts,
// restore the flag, exit Read-write locks
Not all shared data is a queue. The menu is read thousands of times and changed a few times a day. A plain lock would make every reader wait for every other reader. A read write lock allows many readers at once, but a writer gets the lock alone. One caution: with ReentrantReadWriteLock you cannot upgrade a read lock to a write lock; trying deadlocks. Release the read lock first. For a simple map, ConcurrentHashMap may be enough.
rw.readLock().lock();
try { return prices.get(dish); }
finally { rw.readLock().unlock(); }
rw.writeLock().lock();
try { prices.put(dish, price); }
finally { rw.writeLock().unlock(); } The whole flow
Here is the full kitchen. The intake puts order seven into the bounded queue. A pool runs two cook tasks, and whichever cook is free takes the order. When it is ready, the cook records it on a thread safe ready shelf, a ConcurrentHashMap updated with merge. At closing time, the intake sends poison pills, the pool shuts down gracefully, and the totals are printed once every cook has finished.
In LLD interviews
In interviews, these patterns are everywhere. An asynchronous logger puts log lines in a queue drained by one writer thread. A notification service, an order pipeline, an elevator controller taking requests, a web crawler's list of pages to visit: all producer consumer. Mention four things: the queue is bounded, how the pool is sized, what happens when it is full, and how it shuts down.
- async logger: app threads
- put, one writer drains
- notification service
- order or payment pipeline
- elevator request queue
- web crawler: URL frontier
- say: bounded queue, pool
- size, rejection, shutdown
Common mistakes
The common mistakes are about limits and endings. Unbounded queues and a thread per request both fail under load. Catching an interrupted exception and ignoring it stops shutdown from working. Forgetting to shut a pool down keeps the program alive forever. Blocking calls in a small shared pool starve everything else. And an exception inside a submitted task is hidden in its Future until someone calls get.
- • unbounded queues
- • a thread per request
- • swallowing InterruptedException
- • never calling shutdown,
- so the program never exits
- • blocking calls inside a
- small shared pool
- • ignoring Future exceptions