Notes
Requirements
Fix the scope. Create topics and publish messages with a key and a value. Every application that subscribes gets every message, which is pub/sub, but several workers inside one application share the messages, which is a queue. Messages with the same key stay in order. Nothing is lost if a worker crashes: delivery is at least once, with acknowledgements. A slow subscriber must slow the publisher down instead of filling memory.
Clarify the two answers that shape everything. Is ordering needed across the whole topic or only per key, like per order id? Per key, which lets us split a topic into partitions. Which delivery promise? At least once, so consumers must tolerate duplicates. Do consumers pull or are messages pushed? Groups pull, simple subscribers get pushed. And messages stay in a log after reading, so a new consumer can replay them.
- createTopic(name, partitions)
- publish(topic, key, value)
- each subscribing app gets
- every message (fan-out)
- workers in one app share
- the messages (queue)
- order kept per key
- at-least-once delivery, acks
- a slow subscriber slows the
- publisher, not the heap
- many threads at once
- ordering: global or per key?
- per key is enough
- at-most, at-least or exactly
- once? at least, + dedupe
- push or pull?
- pull for groups, push for
- simple subscribers
- keep messages after reading?
- yes, a log with offsets
- full inbox? block the sender
Entities
The entities. The broker holds topics. A topic is split into a fixed number of partitions, and each partition is an append-only log of messages, where a message's position in the log is its offset. On the reading side, a consumer group represents one application, such as billing, and its members are the worker threads or processes that share the work. The group remembers how far it has got in each partition.
Class diagram
Turn them into types. A message is an immutable record with its topic, partition, offset, key and value. A partition is a list we only ever append to, so an offset never changes and any reader can come back to it. Appending and reading are synchronized on the partition, and reads return a copy, so a consumer never sees the list change under it. Keeping data immutable and append-only is what makes replay and redelivery simple.
record Message(String topic,
int partition, long offset,
String key, String value)
final class Partition {
List<Message> log = new ArrayList<>();
synchronized Message append(k, v)
// offset = log.size()
synchronized List<Message>
read(long from, int max) } Strategy: partitioner
Choosing a partition is a Strategy. The default hashes the key and takes it modulo the number of partitions, using floor mod so a negative hash still lands in range. Every event for order one goes to the same partition, and a partition is read in order, so created always comes before paid. There is no order across different keys, and that is the trade we accepted to let partitions be read in parallel. Keyless messages could use round robin.
interface Partitioner {
int partition(String key, int n);
}
KeyHash:
floorMod(key.hashCode(), n)
order-1 always goes to p1, so
created comes before paid
round robin, sticky batches:
new classes, broker unchanged Flow: publish
Walk a publish. Look up the topic, ask the partitioner for the partition, and append, which assigns the next offset. The message is now durable in our log, and pull consumers will find it when they next poll; the publisher does not wait for them. Push subscribers are different: the message is put into each one's inbox, and if an inbox is full, that put blocks. That blocking is our back-pressure, and we will come back to it.
t = topic(name)
p = t.partitions[partitioner
.partition(key, n)]
m = p.append(t.name, key, value)
for (s : t.pushSubs)
s.inbox.put(m) // may block
return m Consumer groups
Consumer groups give us both models with one mechanism. Every group reads the whole topic, so billing and analytics each see every order: that is pub/sub. Inside one group, each partition is owned by exactly one member, so the workers split the load and no message goes to two of them: that is a queue. One owner per partition also keeps per-key order. It means more members than partitions leaves some idle, so pick the partition count for your peak parallelism.
- group billing: worker-a, b
- group analytics: reporter
- every group reads every
- message: pub/sub fan-out
- inside a group each partition
- has one owner: a queue
- members above partitions:
- the extra members idle
Offsets as acks
Offsets replace per-message bookkeeping. For each partition the group keeps two numbers. Position is the next offset to hand to the current owner, and poll advances it. Committed means every message before it has been processed, and the consumer moves it forward after processing: that commit is the acknowledgement. Lag, the end of the log minus committed, tells you how far behind a group is, and it is the first number to put on a dashboard.
long[] position // next to hand out
long[] committed // all before: done
poll(member, max):
read from position[p], advance it
commit(member, m):
committed[p] = max(committed[p],
m.offset + 1)
lag = end offset - committed At-least-once
The delivery promise comes from the order of two steps. If a consumer commits and then processes, a crash in between loses the message: at most once. If it processes and then commits, a crash in between means the next owner gets that message again: at least once. We choose the second, so consumers must be idempotent. In the demo worker-a processes five, commits four and crashes; worker-b receives two, and skips the one already done.
- commit, then process:
- crash loses the message
- = at most once
- process, then commit:
- crash repeats the message
- = at least once
- so consumers must be idempotent:
- skip a seen (partition, offset)
- or use the business key
Rebalance and fencing
When a member joins or leaves, the group rebalances. Partitions are dealt round robin over the sorted members, and every position rewinds to the committed offset, which is exactly how uncommitted work gets redelivered. A generation number counts rebalances. A worker that was presumed dead but comes back must not move offsets it no longer owns, so commit checks the member is the current owner and refuses otherwise. That check is called fencing.
generation++ members sorted: [worker-b] for each partition p: owner[p] = members[p % size] position[p] = committed[p] commit(member, m): member != owner[p]: refuse (a stale worker cannot move offsets it no longer owns)
Back-pressure
Push subscribers get messages delivered by a thread, through an inbox. If the subscriber is slower than the publisher, an unbounded inbox just grows until memory runs out. So the inbox is bounded, and when it is full the publisher's put waits. That is back-pressure: the slow reader slows the writer instead of crashing the process. The alternatives are to drop the newest or oldest message, fine for live prices, or reject and let the caller retry.
- block the publisher
- (our choice: no data lost)
- drop newest or drop oldest
- (metrics, live prices)
- reject with an error
- (caller retries later)
- unbounded queue: hides the
- problem until memory runs out
Bounded blocking queue
Interviewers often ask you to write the bounded blocking queue itself. One reentrant lock guards an array deque, with two conditions. Put waits on not full while the queue is at capacity, adds the item and signals not empty. Take waits on not empty while it is empty, removes the head and signals not full. Always wait in a while loop, not an if, because a thread can wake up without a signal, or another thread can take the slot first.
void put(T x) {
lock(); while (size == cap)
notFull.await();
items.addLast(x); notEmpty.signal();
}
T take() {
lock(); while (items.isEmpty())
notEmpty.await();
x = removeFirst(); notFull.signal();
}
while, not if: spurious wakeups Concurrency
Concurrency. Each partition locks only itself, so publishers to different partitions never wait for each other. Each group synchronizes its poll, commit and rebalance, because owners, positions and committed offsets change together. Topic and group maps are concurrent maps, and push subscriber lists are copy-on-write. In the demo four producers publish two hundred fifty messages each while three consumers drain them, and every message arrives once, with each key in order.
- partition: synchronized append
- and read; reads return copies
- group: synchronized poll,
- commit, join, leave
- topics, groups:
- ConcurrentHashMap
- push subscribers:
- CopyOnWriteArrayList
- test: 4 producers x 250,
- 3 consumers, all 1000 once,
- every key in order
Extensibility
Extensions. Retention deletes old parts of the log by age or size. A message that keeps failing goes to a dead letter topic after a few tries, so it does not block its partition forever. A classic queue acknowledges each message and hides it for a visibility timeout instead of tracking offsets. Exactly once needs the consumer's output and its offset committed together in one transaction. And durability means appending to files on disk.
- retention: drop old segments
- by age or size
- dead letter topic: after N
- failures, park the message
- per-message acks with a
- visibility timeout
- exactly once: idempotent writes
- + offsets in one transaction
- persist: append to segment files
Testing
The full code is below the player. The demo publishes created and paid events for three orders and checks each key stays in order. Billing's worker-a crashes between processing and committing, worker-b takes over, receives two messages and skips one duplicate. A slow pager with an inbox of two holds the publisher back. Finally four producers and three consumers race, and all thousand messages arrive once and in key order.
analytics got 6: order-3:created@p0/0, order-3:paid@p0/1, ... billing: worker-a processed 5, committed 4, then crashed; lag 2 worker-b after rebalance got 2, duplicates skipped 1, lag 0 push: 10 published, inbox never above 2, publisher waited ~15+ ms race: 4 producers x 250, 3 consumers: consumed 1000, lag 0 all checks passed
Follow-ups
Expect follow-ups. Need a total order? Use one partition and accept one reader at a time. More consumers than partitions? The extras sit idle. Adding partitions later changes where keys hash, so ordering breaks once at the switch. A message that always fails goes to a dead letter topic after a few retries. And a group falling behind shows up as growing lag, which is the signal to add members, up to the partition count.
- global order? one partition
- more consumers than partitions?
- extras idle; add partitions
- add partitions later?
- keys move: order breaks once
- poison message?
- retry N times, dead letter
- consumer too slow?
- watch lag, scale the group
Say it like this
The short version. A topic is a set of append-only partitions, and the key picks the partition, so order holds per key. Every consumer group reads everything, and inside a group each partition has one owner, which gives pub/sub and a queue in one design. Consumers process and then commit offsets, so delivery is at least once and consumers deduplicate. Push subscribers have bounded inboxes, so a slow reader slows the writer.
- "A topic is N append-only
- partitions; the key picks one,
- so per-key order holds. Groups
- fan out; in a group each
- partition has one owner.
- Process then commit gives at-
- least-once, so consumers dedupe.
- Push inboxes are bounded."