LOW LEVEL DESIGN · MACHINE CODING · MEDIUM · 10 OF 42

Design a Pub/Sub Message Queue: Low-Level Design in Java

A pub/sub library and message queue designed and coded: topics split into append-only partitions, a key partitioner for per-key ordering, consumer groups that fan out across groups and share work inside one, committed offsets as acks for at-least-once delivery, rebalancing and fencing stale consumers, idempotent consumers, push subscribers behind a bounded blocking queue for back-pressure, and full Java 17 code.

Patterns usedStrategy

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
  42. 42
1 / 19

Press play to hear the narration, or step through at your own pace. The full Java code is below.

Full Java solution

One file, Java 17, 351 lines, with a small main() demo. Save it as PubSub.java, then run javac PubSub.java && java PubSub.

import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeSet;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.locks.Condition;
import java.util.concurrent.locks.ReentrantLock;
import java.util.function.Consumer;

/** A pub/sub broker: partitioned topic logs, consumer groups with committed offsets, push subscribers. */
public final class PubSub {

    // ---------- the model ----------
    record Message(String topic, int partition, long offset, String key, String value) {}

    /** Same key, same partition: that is what gives per-key ordering. */
    interface Partitioner { int partition(String key, int partitions); }

    static final class KeyHash implements Partitioner {
        public int partition(String key, int partitions) { return Math.floorMod(key.hashCode(), partitions); }
    }

    /** An append-only log. Offsets are positions in the list and never change. */
    static final class Partition {
        final int id;
        private final List<Message> log = new ArrayList<>();

        Partition(int id) { this.id = id; }

        synchronized Message append(String topic, String key, String value) {
            Message m = new Message(topic, id, log.size(), key, value);
            log.add(m);
            return m;
        }

        synchronized List<Message> read(long from, int max) {
            int end = (int) Math.min(log.size(), from + max);
            return from >= end ? List.of() : List.copyOf(log.subList((int) from, end));
        }

        synchronized long endOffset() { return log.size(); }
    }

    static final class Topic {
        final String name;
        final Partition[] partitions;
        final List<PushSubscription> pushSubs = new CopyOnWriteArrayList<>();

        Topic(String name, int n) {
            this.name = name;
            partitions = new Partition[n];
            for (int i = 0; i < n; i++) partitions[i] = new Partition(i);
        }
    }

    // ---------- back-pressure: a bounded blocking queue ----------
    /** put() waits while full, take() waits while empty. Two conditions on one lock. */
    static final class BoundedQueue<T> {
        private final ArrayDeque<T> items = new ArrayDeque<>();
        private final int capacity;
        private final ReentrantLock lock = new ReentrantLock();
        private final Condition notFull = lock.newCondition();
        private final Condition notEmpty = lock.newCondition();
        private int maxSeen;

        BoundedQueue(int capacity) { this.capacity = capacity; }

        void put(T item) throws InterruptedException {
            lock.lock();
            try {
                while (items.size() == capacity) notFull.await(); // the producer slows down here
                items.addLast(item);
                maxSeen = Math.max(maxSeen, items.size());
                notEmpty.signal();
            } finally {
                lock.unlock();
            }
        }

        T take() throws InterruptedException {
            lock.lock();
            try {
                while (items.isEmpty()) notEmpty.await();
                T item = items.removeFirst();
                notFull.signal();
                return item;
            } finally {
                lock.unlock();
            }
        }

        int maxSeen() {
            lock.lock();
            try {
                return maxSeen;
            } finally {
                lock.unlock();
            }
        }
    }

    /** Push mode: a bounded inbox and one delivery thread per subscriber. */
    static final class PushSubscription {
        final BoundedQueue<Message> inbox;
        private final Thread deliverer;

        PushSubscription(String name, int capacity, Consumer<Message> handler) {
            inbox = new BoundedQueue<>(capacity);
            deliverer = new Thread(() -> {
                try {
                    while (true) handler.accept(inbox.take());
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            }, name);
        }

        void start() { deliverer.start(); }

        void close() throws InterruptedException {
            deliverer.interrupt();
            deliverer.join();
        }
    }

    // ---------- pull mode: consumer groups with offsets ----------
    /** Every group sees every message; inside a group each partition has exactly one owner. */
    static final class ConsumerGroup {
        final String name;
        private final Topic topic;
        private final long[] committed; // where a new owner starts: everything before is done
        private final long[] position;  // next offset to hand to the current owner
        private final String[] owner;
        private final TreeSet<String> members = new TreeSet<>();
        private int generation;

        ConsumerGroup(String name, Topic topic) {
            this.name = name;
            this.topic = topic;
            int n = topic.partitions.length;
            committed = new long[n];
            position = new long[n];
            owner = new String[n];
        }

        synchronized void join(String member) {
            members.add(member);
            rebalance();
        }

        synchronized void leave(String member) {
            members.remove(member);
            rebalance();
        }

        /** Round-robin partitions over sorted members; uncommitted work is handed out again. */
        private void rebalance() {
            generation++;
            List<String> m = new ArrayList<>(members);
            for (int p = 0; p < owner.length; p++) {
                owner[p] = m.isEmpty() ? null : m.get(p % m.size());
                position[p] = committed[p];
            }
        }

        synchronized List<Integer> assigned(String member) {
            List<Integer> out = new ArrayList<>();
            for (int p = 0; p < owner.length; p++) if (member.equals(owner[p])) out.add(p);
            return out;
        }

        synchronized List<Message> poll(String member, int max) {
            List<Message> out = new ArrayList<>();
            for (int p : assigned(member)) {
                List<Message> batch = topic.partitions[p].read(position[p], max - out.size());
                position[p] += batch.size();
                out.addAll(batch);
                if (out.size() == max) break;
            }
            return out;
        }

        /** Ack: everything up to and including this offset is processed. Stale owners are refused. */
        synchronized boolean commit(String member, Message m) {
            if (!member.equals(owner[m.partition()])) return false;
            committed[m.partition()] = Math.max(committed[m.partition()], m.offset() + 1);
            return true;
        }

        synchronized long lag() {
            long lag = 0;
            for (int p = 0; p < owner.length; p++) lag += topic.partitions[p].endOffset() - committed[p];
            return lag;
        }

        synchronized int generation() { return generation; }
    }

    // ---------- the broker ----------
    private final Map<String, Topic> topics = new ConcurrentHashMap<>();
    private final Map<String, ConsumerGroup> groups = new ConcurrentHashMap<>();
    private final Partitioner partitioner;

    PubSub(Partitioner partitioner) { this.partitioner = partitioner; }

    void createTopic(String name, int partitions) { topics.putIfAbsent(name, new Topic(name, partitions)); }

    private Topic topic(String name) {
        Topic t = topics.get(name);
        if (t == null) throw new IllegalArgumentException("no topic " + name);
        return t;
    }

    /** Append to the key's partition, then hand to push subscribers (blocking if their inbox is full). */
    Message publish(String topicName, String key, String value) throws InterruptedException {
        Topic t = topic(topicName);
        Partition p = t.partitions[partitioner.partition(key, t.partitions.length)];
        Message m = p.append(t.name, key, value);
        for (PushSubscription s : t.pushSubs) s.inbox.put(m);
        return m;
    }

    ConsumerGroup group(String topicName, String groupName) {
        Topic t = topic(topicName);
        return groups.computeIfAbsent(topicName + "/" + groupName, k -> new ConsumerGroup(groupName, t));
    }

    PushSubscription subscribe(String topicName, String name, int capacity, Consumer<Message> handler) {
        PushSubscription s = new PushSubscription(name, capacity, handler);
        topic(topicName).pushSubs.add(s);
        s.start();
        return s;
    }

    // ---------- demo ----------
    static void check(boolean ok, String what) {
        if (!ok) throw new AssertionError(what);
    }

    static String show(Message m) {
        return m.key() + ":" + m.value() + "@p" + m.partition() + "/" + m.offset();
    }

    public static void main(String[] args) throws InterruptedException {
        PubSub broker = new PubSub(new KeyHash());
        broker.createTopic("orders", 3);
        for (String id : List.of("order-1", "order-2", "order-3")) broker.publish("orders", id, "created");
        for (String id : List.of("order-1", "order-2", "order-3")) broker.publish("orders", id, "paid");

        // Two groups: each one gets every message (fan-out).
        ConsumerGroup billing = broker.group("orders", "billing");
        ConsumerGroup analytics = broker.group("orders", "analytics");
        analytics.join("reporter");
        List<Message> all = analytics.poll("reporter", 100);
        System.out.println("analytics got " + all.size() + ": " + all.stream().map(PubSub::show).toList());
        for (String id : List.of("order-1", "order-2", "order-3")) {
            List<String> seq = all.stream().filter(m -> m.key().equals(id)).map(Message::value).toList();
            check(seq.equals(List.of("created", "paid")), "per-key order for " + id);
        }

        // At-least-once: process, then commit. A crash between the two means the message comes again.
        Set<String> processed = new HashSet<>();
        AtomicInteger duplicates = new AtomicInteger();
        Consumer<Message> idempotent = m -> {
            if (!processed.add(m.partition() + "/" + m.offset())) duplicates.incrementAndGet();
        };
        billing.join("worker-a");
        List<Message> got = billing.poll("worker-a", 100);
        for (int i = 0; i < got.size(); i++) {
            idempotent.accept(got.get(i));
            if (i == 4) break; // crash after processing the fifth message, before committing it
            billing.commit("worker-a", got.get(i));
        }
        System.out.println("billing: worker-a processed 5, committed 4, then crashed; lag " + billing.lag());
        billing.join("worker-b");
        billing.leave("worker-a");
        List<Message> again = billing.poll("worker-b", 100);
        again.forEach(idempotent);
        again.forEach(m -> billing.commit("worker-b", m));
        System.out.println("worker-b after rebalance got " + again.size() + ", duplicates skipped "
                + duplicates.get() + ", lag " + billing.lag());
        check(again.size() == 2 && duplicates.get() == 1 && billing.lag() == 0, "redelivery");
        check(!billing.commit("worker-a", again.get(0)), "a stale member cannot commit");

        // Back-pressure: a slow push subscriber with an inbox of 2 slows the publisher down.
        broker.createTopic("alerts", 1);
        List<String> delivered = new CopyOnWriteArrayList<>();
        PushSubscription pager = broker.subscribe("alerts", "pager", 2, m -> {
            try {
                Thread.sleep(3);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
            delivered.add(m.value());
        });
        long t0 = System.nanoTime();
        for (int i = 1; i <= 10; i++) broker.publish("alerts", "disk", "alert-" + i);
        long blockedMs = (System.nanoTime() - t0) / 1_000_000;
        while (delivered.size() < 10) Thread.sleep(1);
        pager.close();
        System.out.println("push: 10 published, inbox never above " + pager.inbox.maxSeen()
                + ", publisher waited ~" + (blockedMs >= 15 ? "15+" : String.valueOf(blockedMs)) + " ms");
        check(pager.inbox.maxSeen() <= 2 && delivered.get(9).equals("alert-10"), "back-pressure");

        // Four producers, three consumers in one group: every message exactly once here, in key order.
        broker.createTopic("events", 3);
        ConsumerGroup audit = broker.group("events", "audit");
        String[] names = {"auditor-1", "auditor-2", "auditor-3"};
        Arrays.stream(names).forEach(audit::join);
        AtomicInteger consumed = new AtomicInteger();
        Map<String, Integer> lastSeq = new ConcurrentHashMap<>();
        List<Thread> threads = new ArrayList<>();
        for (int p = 0; p < 4; p++) {
            String key = "producer-" + p;
            threads.add(new Thread(() -> {
                try {
                    for (int i = 0; i < 250; i++) broker.publish("events", key, String.valueOf(i));
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            }));
        }
        for (String member : names) {
            threads.add(new Thread(() -> {
                while (consumed.get() < 1000) {
                    for (Message m : audit.poll(member, 50)) {
                        int seq = Integer.parseInt(m.value());
                        Integer prev = lastSeq.put(m.key(), seq);
                        if (prev != null && prev != seq - 1) throw new AssertionError("out of order " + m);
                        audit.commit(member, m);
                        consumed.incrementAndGet();
                    }
                    Thread.onSpinWait();
                }
            }));
        }
        threads.forEach(Thread::start);
        for (Thread t : threads) t.join(10_000);
        System.out.println("race: 4 producers x 250, 3 consumers: consumed " + consumed.get()
                + ", lag " + audit.lag() + ", rebalances " + audit.generation());
        check(consumed.get() == 1000 && audit.lag() == 0, "exactly the published messages");
        check(lastSeq.values().stream().allMatch(v -> v == 249), "every key read to the end, in order");
        System.out.println("all checks passed");
    }
}

Notes

Requirements

Fix the scope. Create topics and publish messages with a key and a value. Every application that subscribes gets every message, which is pub/sub, but several workers inside one application share the messages, which is a queue. Messages with the same key stay in order. Nothing is lost if a worker crashes: delivery is at least once, with acknowledgements. A slow subscriber must slow the publisher down instead of filling memory.

Clarify the two answers that shape everything. Is ordering needed across the whole topic or only per key, like per order id? Per key, which lets us split a topic into partitions. Which delivery promise? At least once, so consumers must tolerate duplicates. Do consumers pull or are messages pushed? Groups pull, simple subscribers get pushed. And messages stay in a log after reading, so a new consumer can replay them.

Requirements
  • createTopic(name, partitions)
  • publish(topic, key, value)
  • each subscribing app gets
  • every message (fan-out)
  • workers in one app share
  • the messages (queue)
  • order kept per key
  • at-least-once delivery, acks
  • a slow subscriber slows the
  • publisher, not the heap
  • many threads at once
Clarify first
  • ordering: global or per key?
  • per key is enough
  • at-most, at-least or exactly
  • once? at least, + dedupe
  • push or pull?
  • pull for groups, push for
  • simple subscribers
  • keep messages after reading?
  • yes, a log with offsets
  • full inbox? block the sender

Entities

The entities. The broker holds topics. A topic is split into a fixed number of partitions, and each partition is an append-only log of messages, where a message's position in the log is its offset. On the reading side, a consumer group represents one application, such as billing, and its members are the worker threads or processes that share the work. The group remembers how far it has got in each partition.

Class diagram

Turn them into types. A message is an immutable record with its topic, partition, offset, key and value. A partition is a list we only ever append to, so an offset never changes and any reader can come back to it. Appending and reading are synchronized on the partition, and reads return a copy, so a consumer never sees the list change under it. Keeping data immutable and append-only is what makes replay and redelivery simple.

Log and message
record Message(String topic,
    int partition, long offset,
    String key, String value)

final class Partition {
  List<Message> log = new ArrayList<>();
  synchronized Message append(k, v)
    // offset = log.size()
  synchronized List<Message>
    read(long from, int max) }

Strategy: partitioner

Choosing a partition is a Strategy. The default hashes the key and takes it modulo the number of partitions, using floor mod so a negative hash still lands in range. Every event for order one goes to the same partition, and a partition is read in order, so created always comes before paid. There is no order across different keys, and that is the trade we accepted to let partitions be read in parallel. Keyless messages could use round robin.

Partitioner
interface Partitioner {
  int partition(String key, int n);
}
KeyHash:
  floorMod(key.hashCode(), n)

order-1 always goes to p1, so
  created comes before paid
round robin, sticky batches:
  new classes, broker unchanged

Flow: publish

Walk a publish. Look up the topic, ask the partitioner for the partition, and append, which assigns the next offset. The message is now durable in our log, and pull consumers will find it when they next poll; the publisher does not wait for them. Push subscribers are different: the message is put into each one's inbox, and if an inbox is full, that put blocks. That blocking is our back-pressure, and we will come back to it.

publish(topic, key, value)
t = topic(name)
p = t.partitions[partitioner
      .partition(key, n)]
m = p.append(t.name, key, value)
for (s : t.pushSubs)
  s.inbox.put(m)   // may block
return m

Consumer groups

Consumer groups give us both models with one mechanism. Every group reads the whole topic, so billing and analytics each see every order: that is pub/sub. Inside one group, each partition is owned by exactly one member, so the workers split the load and no message goes to two of them: that is a queue. One owner per partition also keeps per-key order. It means more members than partitions leaves some idle, so pick the partition count for your peak parallelism.

Pub/sub and queue in one
  • group billing: worker-a, b
  • group analytics: reporter
  • every group reads every
  • message: pub/sub fan-out
  • inside a group each partition
  • has one owner: a queue
  • members above partitions:
  • the extra members idle

Offsets as acks

Offsets replace per-message bookkeeping. For each partition the group keeps two numbers. Position is the next offset to hand to the current owner, and poll advances it. Committed means every message before it has been processed, and the consumer moves it forward after processing: that commit is the acknowledgement. Lag, the end of the log minus committed, tells you how far behind a group is, and it is the first number to put on a dashboard.

position and committed
long[] position   // next to hand out
long[] committed  // all before: done

poll(member, max):
  read from position[p], advance it
commit(member, m):
  committed[p] = max(committed[p],
                     m.offset + 1)
lag = end offset - committed

At-least-once

The delivery promise comes from the order of two steps. If a consumer commits and then processes, a crash in between loses the message: at most once. If it processes and then commits, a crash in between means the next owner gets that message again: at least once. We choose the second, so consumers must be idempotent. In the demo worker-a processes five, commits four and crashes; worker-b receives two, and skips the one already done.

The order of two steps
  • commit, then process:
  • crash loses the message
  • = at most once
  • process, then commit:
  • crash repeats the message
  • = at least once
  • so consumers must be idempotent:
  • skip a seen (partition, offset)
  • or use the business key

Rebalance and fencing

When a member joins or leaves, the group rebalances. Partitions are dealt round robin over the sorted members, and every position rewinds to the committed offset, which is exactly how uncommitted work gets redelivered. A generation number counts rebalances. A worker that was presumed dead but comes back must not move offsets it no longer owns, so commit checks the member is the current owner and refuses otherwise. That check is called fencing.

rebalance()
generation++
members sorted: [worker-b]
for each partition p:
  owner[p] = members[p % size]
  position[p] = committed[p]

commit(member, m):
  member != owner[p]: refuse
  (a stale worker cannot move
   offsets it no longer owns)

Back-pressure

Push subscribers get messages delivered by a thread, through an inbox. If the subscriber is slower than the publisher, an unbounded inbox just grows until memory runs out. So the inbox is bounded, and when it is full the publisher's put waits. That is back-pressure: the slow reader slows the writer instead of crashing the process. The alternatives are to drop the newest or oldest message, fine for live prices, or reject and let the caller retry.

Choices when the inbox is full
  • block the publisher
  • (our choice: no data lost)
  • drop newest or drop oldest
  • (metrics, live prices)
  • reject with an error
  • (caller retries later)
  • unbounded queue: hides the
  • problem until memory runs out

Bounded blocking queue

Interviewers often ask you to write the bounded blocking queue itself. One reentrant lock guards an array deque, with two conditions. Put waits on not full while the queue is at capacity, adds the item and signals not empty. Take waits on not empty while it is empty, removes the head and signals not full. Always wait in a while loop, not an if, because a thread can wake up without a signal, or another thread can take the slot first.

BoundedQueue<T>
void put(T x) {
  lock(); while (size == cap)
            notFull.await();
  items.addLast(x); notEmpty.signal();
}
T take() {
  lock(); while (items.isEmpty())
            notEmpty.await();
  x = removeFirst(); notFull.signal();
}
while, not if: spurious wakeups

Concurrency

Concurrency. Each partition locks only itself, so publishers to different partitions never wait for each other. Each group synchronizes its poll, commit and rebalance, because owners, positions and committed offsets change together. Topic and group maps are concurrent maps, and push subscriber lists are copy-on-write. In the demo four producers publish two hundred fifty messages each while three consumers drain them, and every message arrives once, with each key in order.

Thread safety
  • partition: synchronized append
  • and read; reads return copies
  • group: synchronized poll,
  • commit, join, leave
  • topics, groups:
  • ConcurrentHashMap
  • push subscribers:
  • CopyOnWriteArrayList
  • test: 4 producers x 250,
  • 3 consumers, all 1000 once,
  • every key in order

Extensibility

Extensions. Retention deletes old parts of the log by age or size. A message that keeps failing goes to a dead letter topic after a few tries, so it does not block its partition forever. A classic queue acknowledges each message and hides it for a visibility timeout instead of tracking offsets. Exactly once needs the consumer's output and its offset committed together in one transaction. And durability means appending to files on disk.

Extension points
  • retention: drop old segments
  • by age or size
  • dead letter topic: after N
  • failures, park the message
  • per-message acks with a
  • visibility timeout
  • exactly once: idempotent writes
  • + offsets in one transaction
  • persist: append to segment files

Testing

The full code is below the player. The demo publishes created and paid events for three orders and checks each key stays in order. Billing's worker-a crashes between processing and committing, worker-b takes over, receives two messages and skips one duplicate. A slow pager with an inbox of two holds the publisher back. Finally four producers and three consumers race, and all thousand messages arrive once and in key order.

Run it
analytics got 6: order-3:created@p0/0,
  order-3:paid@p0/1, ...
billing: worker-a processed 5,
  committed 4, then crashed; lag 2
worker-b after rebalance got 2,
  duplicates skipped 1, lag 0
push: 10 published, inbox never
  above 2, publisher waited ~15+ ms
race: 4 producers x 250, 3 consumers:
  consumed 1000, lag 0
all checks passed

Follow-ups

Expect follow-ups. Need a total order? Use one partition and accept one reader at a time. More consumers than partitions? The extras sit idle. Adding partitions later changes where keys hash, so ordering breaks once at the switch. A message that always fails goes to a dead letter topic after a few retries. And a group falling behind shows up as growing lag, which is the signal to add members, up to the partition count.

Follow-ups to expect
  • global order? one partition
  • more consumers than partitions?
  • extras idle; add partitions
  • add partitions later?
  • keys move: order breaks once
  • poison message?
  • retry N times, dead letter
  • consumer too slow?
  • watch lag, scale the group

Say it like this

The short version. A topic is a set of append-only partitions, and the key picks the partition, so order holds per key. Every consumer group reads everything, and inside a group each partition has one owner, which gives pub/sub and a queue in one design. Consumers process and then commit offsets, so delivery is at least once and consumers deduplicate. Push subscribers have bounded inboxes, so a slow reader slows the writer.

In the interview
  • "A topic is N append-only
  • partitions; the key picks one,
  • so per-key order holds. Groups
  • fan out; in a group each
  • partition has one owner.
  • Process then commit gives at-
  • least-once, so consumers dedupe.
  • Push inboxes are bounded."

Interview questions

How do you keep messages in order?

Only per partition. A partitioner hashes the message key to a partition, so all messages for one key land in the same append-only log, and within a group only one consumer reads a partition. There is no order across keys, which is what lets partitions be processed in parallel.

How do consumer groups give both pub/sub and queue behaviour?

Each group reads every message of the topic, so different applications each get a copy (pub/sub). Inside a group each partition is owned by one member, so members share the messages and none is processed twice by the group (queue).

How do you get at-least-once delivery?

The consumer processes a message and only then commits its offset. If it crashes in between, the partition is reassigned and the new owner starts from the last committed offset, so the message is delivered again. Consumers must therefore be idempotent.

What is the difference between position and committed offset?

Position is the next offset to hand to the current owner and moves on every poll. Committed is the offset before which everything has been processed and moves only on commit. After a rebalance the position is reset to committed, which is how unacknowledged work is redelivered.

How do you stop a consumer that was presumed dead from corrupting offsets?

Fence it. On rebalance the group records new owners and bumps a generation; commit refuses any member that is not the current owner of the partition, so a returning stale consumer cannot move offsets.

How does a bounded blocking queue provide back-pressure?

put waits on a not-full condition while the queue is at capacity, and take waits on not-empty while it is empty. A slow consumer leaves the queue full, so the producer blocks instead of piling messages into memory. Waits are in while loops to handle spurious wakeups.

How would you make processing effectively exactly once?

Make the consumer idempotent, for example by recording processed partition and offset or a business id, or write the result and the committed offset in one transaction so they succeed or fail together.

What happens with more consumers than partitions?

Each partition has one owner per group, so the extra consumers get no partitions and sit idle. The partition count caps a group's parallelism, so choose it for peak load; adding partitions later remaps keys and briefly breaks per-key ordering.