LOW-LEVEL DESIGN · CONCURRENCY FOR LLD · CONCURRENCY · 46 OF 48

Producer-Consumer and Thread Pools in Java: BlockingQueue Guide

The concurrency patterns behind most LLD answers, shown with a restaurant kitchen: producer-consumer with a BlockingQueue, bounded queues and back-pressure, ExecutorService thread pools and how to size them, rejection policies, poison pills, graceful shutdown with shutdown and awaitTermination, and read-write locks for read-heavy caches.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
  42. 42
  43. 43
  44. 44
  45. 45
  46. 46
  47. 47
  48. 48
1 / 15

Press play to hear the narration, or step through at your own pace. The Java example is below.

Java example

One file, Java 17, 167 lines, with a small main() demo. Save it as KitchenQueueDemo.java, then run javac KitchenQueueDemo.java && java KitchenQueueDemo.

import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.Callable;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.locks.ReentrantReadWriteLock;

/** A restaurant kitchen: bounded queue, pool of cooks, poison pills, clean shutdown, read-write lock. */
public class KitchenQueueDemo {

    record Order(int id, String dish) { }

    static final Order STOP = new Order(-1, "stop"); // poison pill

    public static void main(String[] args) throws Exception {
        unboundedProblem();
        backPressure();
        producerConsumer();
        callerRuns();
        readWriteLock();
    }

    // ---------- 1. The problem: an unbounded queue never says no ----------
    static void unboundedProblem() {
        BlockingQueue<Order> q = new LinkedBlockingQueue<>(); // no limit
        for (int i = 0; i < 100_000; i++) {
            q.offer(new Order(i, "dosa"));                     // nobody is consuming
        }
        System.out.println("Unbounded queue silently holds " + q.size() + " waiting orders");
    }

    // ---------- 2. A bounded queue pushes back ----------
    static void backPressure() throws InterruptedException {
        BlockingQueue<Order> q = new ArrayBlockingQueue<>(2);
        System.out.println("offer 1: " + q.offer(new Order(1, "idli")));
        System.out.println("offer 2: " + q.offer(new Order(2, "vada")));
        System.out.println("offer 3: " + q.offer(new Order(3, "poha")) + " (full: tell the customer)");
        boolean late = q.offer(new Order(4, "upma"), 20, TimeUnit.MILLISECONDS);
        System.out.println("offer 4 with 20 ms wait: " + late);
    }

    // ---------- 3. Producer-consumer on a fixed pool, stopped with poison pills ----------
    static void producerConsumer() throws Exception {
        BlockingQueue<Order> q = new ArrayBlockingQueue<>(4);
        Map<String, Integer> readyShelf = new ConcurrentHashMap<>();
        AtomicInteger cooked = new AtomicInteger();
        int cooks = 2;
        ExecutorService pool = Executors.newFixedThreadPool(cooks + 1);
        List<Future<?>> jobs = new ArrayList<>();

        String[] dishes = {"paneer wrap", "dal rice", "masala dosa"};
        Callable<Void> intake = () -> {
            for (int i = 1; i <= 12; i++) {
                q.put(new Order(i, dishes[i % dishes.length])); // waits while the queue is full
            }
            for (int c = 0; c < cooks; c++) {
                q.put(STOP);                                    // one pill per cook
            }
            return null;
        };
        Callable<Void> cook = () -> {
            while (true) {
                Order o = q.take();                             // sleeps while empty
                if (o == STOP) {
                    return null;
                }
                Thread.sleep(3);                                // pretend to cook
                readyShelf.merge(o.dish(), 1, Integer::sum);
                cooked.incrementAndGet();
            }
        };
        jobs.add(pool.submit(intake));
        for (int c = 0; c < cooks; c++) {
            jobs.add(pool.submit(cook));
        }
        for (Future<?> f : jobs) {
            f.get(5, TimeUnit.SECONDS);                         // surfaces any exception
        }
        shutdownGracefully(pool);
        System.out.println("Cooked " + cooked.get() + " orders: " + new TreeMap<>(readyShelf));
    }

    static void shutdownGracefully(ExecutorService pool) throws InterruptedException {
        pool.shutdown();                                        // no new tasks
        if (!pool.awaitTermination(5, TimeUnit.SECONDS)) {
            pool.shutdownNow();                                 // interrupt stragglers
        }
        System.out.println("Pool terminated: " + pool.isTerminated());
    }

    // ---------- 4. Bounded ThreadPoolExecutor: when full, the caller runs the task ----------
    static void callerRuns() throws InterruptedException {
        ThreadPoolExecutor pool = new ThreadPoolExecutor(2, 2, 0, TimeUnit.SECONDS,
                new ArrayBlockingQueue<>(2), new ThreadPoolExecutor.CallerRunsPolicy());
        AtomicInteger done = new AtomicInteger();
        for (int i = 0; i < 10; i++) {
            pool.execute(() -> {
                try {
                    Thread.sleep(2);
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                    return;
                }
                done.incrementAndGet();
            });
        }
        shutdownGracefully(pool);
        System.out.println("Tasks finished with CallerRunsPolicy: " + done.get() + " of 10, none dropped");
    }

    // ---------- 5. Read-write lock: many readers together, one writer alone ----------
    static final class MenuCache {
        private final ReentrantReadWriteLock rw = new ReentrantReadWriteLock();
        private final Map<String, Integer> prices = new java.util.HashMap<>();

        int price(String dish) {
            rw.readLock().lock();
            try {
                return prices.getOrDefault(dish, 0);
            } finally {
                rw.readLock().unlock();
            }
        }

        void setPrice(String dish, int rupees) {
            rw.writeLock().lock();
            try {
                prices.put(dish, rupees);
            } finally {
                rw.writeLock().unlock();
            }
        }
    }

    static void readWriteLock() throws Exception {
        MenuCache menu = new MenuCache();
        menu.setPrice("masala dosa", 90);
        ExecutorService pool = Executors.newFixedThreadPool(4);
        AtomicInteger reads = new AtomicInteger();
        List<Future<?>> jobs = new ArrayList<>();
        for (int r = 0; r < 3; r++) {
            jobs.add(pool.submit(() -> {
                for (int k = 0; k < 1000; k++) {
                    if (menu.price("masala dosa") > 0) {
                        reads.incrementAndGet();
                    }
                }
            }));
        }
        jobs.add(pool.submit(() -> menu.setPrice("masala dosa", 110)));
        for (Future<?> f : jobs) {
            f.get(5, TimeUnit.SECONDS);
        }
        shutdownGracefully(pool);
        System.out.println("Reads served: " + reads.get() + ", final price: " + menu.price("masala dosa"));
    }
}

Notes

The problem

Two naive designs come up a lot. The first starts a new thread for every order. At lunch that means hundreds of threads, each with its own stack, fighting for a few CPU cores, until the process runs out of memory. The second shares a plain list and spins in a loop waiting for work, which burns a whole core doing nothing, and the list is not even thread safe.

Naive attempts
// 1: a thread per order
new Thread(() -> cook(order)).start();
// 500 orders = 500 threads

// 2: a shared list + busy loop
while (orders.isEmpty()) { }  // burns CPU
cook(orders.remove(0));  // not safe

Producer-consumer

The producer consumer pattern puts a queue between the side that creates work and the side that does it. The order intake is the producer: it puts orders into the queue. Cooks are consumers: they take orders out. Neither side knows about the other. You can add cooks at lunch without touching intake code, and a slow cook does not slow down taking orders, up to the queue's limit.

BlockingQueue

Java gives you this queue ready made: BlockingQueue. Take puts the cook to sleep while the queue is empty and wakes it the moment an order arrives, so no CPU is wasted. Put does the opposite: if the queue is full, the producer sleeps until there is room. The queue handles all the locking inside. For a deadline, offer and poll accept a timeout and return false or null instead of waiting forever.

BlockingQueue
BlockingQueue<Order> q =
    new ArrayBlockingQueue<>(4);

// producer
q.put(order);     // waits if full

// consumer
Order o = q.take(); // waits if empty

offer(o, 50, MILLISECONDS) // or give up

Back-pressure

Always give the queue a limit. An unbounded queue looks friendly, but when producers are faster than consumers, it grows until memory runs out, and every order waits longer and longer. A bounded queue gives back pressure. When it is full, the producer must wait, fail fast, or tell the customer the kitchen is busy. A clear no in one second beats a silent forty minute wait.

Thread pools

A thread pool is producer consumer packaged for you. An ExecutorService keeps a fixed set of worker threads and an internal queue of tasks. You submit a task and get back a Future, which you can use later to read the result or the exception. Threads are created once and reused, so a burst of a thousand tasks still uses only the threads you allowed.

ExecutorService
ExecutorService pool =
    Executors.newFixedThreadPool(2);

Future<Integer> f =
    pool.submit(() -> cook(order));
int minutes = f.get();

// a pool IS a queue + workers

Sizing a pool

How many threads? For work that keeps the CPU busy, like resizing images, about one thread per core; more just adds switching. For work that mostly waits, like calling a payment service, you can use more, roughly cores times one plus the ratio of wait time to work time. Then measure and tune. Keep slow work in its own pool, so a stuck payment provider cannot starve the fast menu lookups.

How many threads?
  • CPU-bound (resize images):
  • threads about = cores
  • IO-bound (call payment API):
  • threads = cores x
  • (1 + wait time / work time)
  • measure, then tune
  • separate pools for slow and
  • fast work (bulkheads)

Rejection policies

One trap: the factory method for a fixed thread pool uses an unbounded queue. For production, build a ThreadPoolExecutor yourself with a bounded queue and a rejection policy. The default policy throws an exception. Caller runs policy makes the submitting thread do the task itself, which naturally slows the producer, a neat form of back pressure. The discard policies drop work, which is only fine for things like metrics.

ThreadPoolExecutor
new ThreadPoolExecutor(
    2, 2, 0, SECONDS,
    new ArrayBlockingQueue<>(4),
    new CallerRunsPolicy());

AbortPolicy: throw (default)
CallerRunsPolicy: caller runs it,
  which slows the producer
Discard / DiscardOldest: drop

Stopping consumers

How does a consumer know the day is over? A common trick is the poison pill: a special marker object put into the queue after the last real order. A cook that takes the marker stops its loop and exits. Put one marker per consumer, so every cook gets one. Because the queue is first in first out, all real orders ahead of the markers are cooked first. Nothing is lost.

Poison pill
static final Order STOP =
    new Order(-1, "stop");

while (true) {
  Order o = q.take();
  if (o == STOP) return;
  cook(o);
}
// producer: one STOP per consumer

Graceful shutdown

For a pool, graceful shutdown is three steps. Call shutdown: no new tasks are accepted, but queued and running ones continue. Then await termination with a time limit. If the pool is still busy after that, call shutdown now, which interrupts the workers and returns the tasks that never started. Workers should respond to interruption by tidying up and exiting, not by swallowing the exception.

Shut down cleanly
pool.shutdown();  // no new tasks
if (!pool.awaitTermination(
        5, TimeUnit.SECONDS)) {
    pool.shutdownNow(); // interrupt
}

// workers: handle interrupts,
// restore the flag, exit

Read-write locks

Not all shared data is a queue. The menu is read thousands of times and changed a few times a day. A plain lock would make every reader wait for every other reader. A read write lock allows many readers at once, but a writer gets the lock alone. One caution: with ReentrantReadWriteLock you cannot upgrade a read lock to a write lock; trying deadlocks. Release the read lock first. For a simple map, ConcurrentHashMap may be enough.

ReentrantReadWriteLock
rw.readLock().lock();
try { return prices.get(dish); }
finally { rw.readLock().unlock(); }

rw.writeLock().lock();
try { prices.put(dish, price); }
finally { rw.writeLock().unlock(); }

The whole flow

Here is the full kitchen. The intake puts order seven into the bounded queue. A pool runs two cook tasks, and whichever cook is free takes the order. When it is ready, the cook records it on a thread safe ready shelf, a ConcurrentHashMap updated with merge. At closing time, the intake sends poison pills, the pool shuts down gracefully, and the totals are printed once every cook has finished.

In LLD interviews

In interviews, these patterns are everywhere. An asynchronous logger puts log lines in a queue drained by one writer thread. A notification service, an order pipeline, an elevator controller taking requests, a web crawler's list of pages to visit: all producer consumer. Mention four things: the queue is bounded, how the pool is sized, what happens when it is full, and how it shuts down.

Where it fits
  • async logger: app threads
  • put, one writer drains
  • notification service
  • order or payment pipeline
  • elevator request queue
  • web crawler: URL frontier
  • say: bounded queue, pool
  • size, rejection, shutdown

Common mistakes

The common mistakes are about limits and endings. Unbounded queues and a thread per request both fail under load. Catching an interrupted exception and ignoring it stops shutdown from working. Forgetting to shut a pool down keeps the program alive forever. Blocking calls in a small shared pool starve everything else. And an exception inside a submitted task is hidden in its Future until someone calls get.

Avoid these
  • • unbounded queues
  • • a thread per request
  • • swallowing InterruptedException
  • • never calling shutdown,
  • so the program never exits
  • • blocking calls inside a
  • small shared pool
  • • ignoring Future exceptions

Interview questions

What is the producer-consumer pattern?

Producers put work items into a shared queue and consumers take them out and process them. The queue decouples the two sides, smooths out bursts and lets you scale producers and consumers independently.

Why use a BlockingQueue instead of a list with locks?

BlockingQueue already handles locking and waiting: take blocks while empty and put blocks while full, without busy loops. It also offers timed offer and poll, which makes timeouts and back-pressure easy.

What is back-pressure?

A way for a slow consumer to push back on fast producers. With a bounded queue, producers must wait, fail fast or shed load when the queue is full, instead of letting the queue grow until memory runs out.

How do you size a thread pool?

For CPU-bound work, roughly the number of cores. For IO-bound work, more threads, about cores times one plus wait time over compute time. Then measure under load and keep slow dependencies in separate pools.

What is wrong with Executors.newFixedThreadPool in production?

It uses an unbounded LinkedBlockingQueue, so tasks can pile up without limit when workers are slow. Creating a ThreadPoolExecutor with a bounded queue and an explicit rejection policy gives you control.

How do you shut down an ExecutorService gracefully?

Call shutdown so no new tasks are accepted, then awaitTermination with a timeout. If it does not finish, call shutdownNow to interrupt workers. Tasks should handle interruption by cleaning up and exiting.

What is a poison pill?

A special marker item placed in the queue to tell a consumer to stop. Since the queue is FIFO, all real items before it are processed first. With several consumers you send one marker per consumer.

When does a read-write lock help?

When data is read much more often than written. Many readers can hold the read lock together while a writer gets exclusive access. It adds overhead, so for simple maps a ConcurrentHashMap or immutable snapshots may be better.

What does CallerRunsPolicy do?

When the pool and its queue are full, the thread that submitted the task runs it itself. That slows the producer down naturally, acting as back-pressure without dropping work.