Start practising

SYSTEM DESIGN · TECHNOLOGIES · 9 OF 12

ZooKeeper and etcd: Small, Strongly Consistent Stores for Coordination

What coordination services are for and why they sit beside your real databases: ZooKeeper znodes and etcd keys, consensus with ZAB and Raft, why ensembles have 3 or 5 members, sessions and leases, watches, leader election, locks with fencing tokens, service discovery and configuration, the size and throughput limits, and how to use them in an interview.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
1 / 16

Press play to hear the narration, or step through at your own pace.

Notes

What they are

Both are small, strongly consistent, replicated key-value stores built for coordination, not for application data. ZooKeeper is the older one, written in Java, using a protocol called ZAB, and it sits under systems like HBase and older Kafka. etcd is written in Go, uses Raft, and is the brain of every Kubernetes cluster. Consul from HashiCorp plays a similar role.

In one line
  • small, strongly consistent,
  • replicated key-value store
  • for coordination, not data
  • ZooKeeper: Java, ZAB protocol
  • (Hadoop, HBase, older Kafka)
  • etcd: Go, Raft protocol
  • (Kubernetes, many Go apps)
  • similar: Consul (Raft too)

Data model

The data models differ a little. ZooKeeper is a tree of nodes called znodes, like a file system, each holding a small value. A znode can be ephemeral, deleted automatically when its client's session dies, and sequential, with a counter appended to its name. etcd is a flat, sorted key-value space where you read ranges by prefix, every change bumps a global revision number, and keys can be attached to a lease.

Two data models
ZooKeeper: tree of znodes
  /app/config          "v=42"
  /app/leader          "host-3"
  /app/locks/lock-0007 (ephemeral,
                        sequential)

etcd: flat, ordered keys
  /app/config  rev 1081
  /app/leader  lease 7f3a
  range reads by prefix

Consensus

Every write goes through the leader. It appends the write to its log, sends it to the followers, and once a majority have it, the write is committed and applied in the same order on every member. If the leader dies, the others elect a new one. During a network split, only the side with a majority can accept writes, so two halves can never disagree. Concepts episode 34 covers Raft step by step.

How a write commits
  • 1. client sends write
  • (followers forward it)
  • 2. leader appends to its log,
  • sends to followers
  • 3. majority has it → commit
  • 4. apply in the same order
  • everywhere
  • leader dies → election;
  • only a majority side can
  • make progress

Ensemble size

How many members? A write needs a majority, so three members survive one failure and five survive two. Four is no better than three, because its majority is three, so it still tolerates only one loss. Seven survives three but every write waits for four acknowledgements, so it is slower. Most deployments use three or five, spread across availability zones.

Why 3 or 5
members  majority  can lose
  1         1         0
  3         2         1
  4         3         1   (no gain)
  5         3         2
  7         4         3   (slower)

more members = slower writes
spread across 3 zones

Sessions + leases

The feature that makes coordination work is tying data to liveness. In ZooKeeper a client holds a session kept alive by heartbeats; if it stops, its ephemeral znodes are deleted. In etcd you grant a lease, say ten seconds, attach keys to it and keep it alive; if the client crashes, the lease expires and the keys disappear. That is how a dead leader or a dead lock holder is cleaned up automatically.

Sessions and leases
  • ZooKeeper session: client
  • heartbeats; on timeout its
  • ephemeral znodes are deleted
  • etcd lease: grant(10 s),
  • attach keys, keepalive;
  • lease expires → keys deleted
  • this is how "who is alive"
  • and "who holds the lock"
  • clean up after crashes

Watches

Clients can watch a key or prefix and be notified when it changes, instead of polling. Change the config and every service watching it hears about it within milliseconds. Classic ZooKeeper watches fire once and must be set again; newer versions add persistent watches. etcd watches are a stream starting from a revision, so a client that reconnects can resume exactly where it left off without missing a change.

Watches
  • client watches a key or prefix
  • → notified when it changes
  • ZooKeeper: one-shot watch,
  • re-register after it fires
  • (persistent watches in 3.6+)
  • etcd: a stream from a
  • revision; never misses one
  • config push, membership,
  • leader changes

Leader election

Leader election is a few lines on top of this. In ZooKeeper, each candidate creates an ephemeral, sequential znode under an election path. The lowest number is leader, and every other candidate watches only the one just ahead of it, which avoids a stampede. In etcd, candidates try to create a key with their lease only if it does not exist yet; the winner leads and the rest watch. When the leader dies, its key vanishes and the next one takes over.

Election recipes
  • ZooKeeper: each candidate
  • creates an ephemeral +
  • sequential znode; lowest
  • number leads; each watches
  • the one just before it
  • etcd: create /election/x with
  • a lease only if absent
  • (txn); others watch it
  • leader dies → key gone → next

Locks + fencing

Locks have a famous trap. A client holds a lock, then pauses for a long garbage collection. Its lease expires, another client takes the lock, and then the first wakes up and writes, believing it still holds it. The fix is a fencing token: each lock grant comes with an increasing number, the transaction id in ZooKeeper or the revision in etcd, and the storage rejects any write carrying an older number than it has already seen.

Locks need fencing
  • problem: holder pauses (GC),
  • lease expires, a second
  • client takes the lock, then
  • the first wakes and writes
  • fix: every grant returns an
  • increasing number
  • (zxid / etcd revision)
  • storage rejects any write
  • with an older token
  • (Concepts episode 31)

Reads

Reads are a subtle interview point. etcd reads are linearizable by default: the leader confirms it is still the leader before answering, so you always see the latest committed value; you can opt into faster local reads that may be slightly stale. ZooKeeper serves reads from whichever member you are connected to, which may be a little behind, and you call sync first when you need the latest. Writes are always ordered through the leader.

Read consistency
  • etcd: linearizable by default
  • (leader confirms it is still
  • leader); serializable reads
  • are local and faster
  • ZooKeeper: reads served by
  • the member you are on,
  • possibly behind; call sync()
  • first for the latest
  • writes always linearizable

Real uses

Where you meet them. Kubernetes stores every pod, service and secret in etcd, and controllers watch for changes. Patroni uses etcd to decide which Postgres replica becomes primary. HBase and Solr rely on ZooKeeper. A common pattern for service discovery: each instance registers itself under a prefix with a lease, and clients watch the prefix. They also hold shard maps, feature flags and which instance runs a scheduled job.

Where you meet them
  • Kubernetes: every pod,
  • service and secret in etcd
  • Patroni: Postgres failover
  • HBase, Solr, older Kafka:
  • ZooKeeper
  • service discovery: register
  • with a lease, watch prefix
  • feature flags, shard maps,
  • config, cron leadership

Where they hurt

Where they hurt. They hold small data: about a megabyte per znode in ZooKeeper, a request size of about one and a half megabytes in etcd, and an etcd database quota of two gigabytes by default. Writes are limited to thousands, maybe low tens of thousands, a second, because each one needs a disk flush and a majority round trip. Slow disks make the whole cluster slow. Lose the majority and writes stop entirely.

Limits and pain
  • • small values only:
  • ZK ~1 MB/znode, etcd 1.5 MB
  • per request
  • • etcd quota 2 GB default
  • (8 GB suggested max)
  • • writes: thousands to low
  • tens of thousands per second
  • • latency = disk fsync +
  • network round trip
  • • majority lost = no writes

Use it when

Use them for small, critical facts that every node must agree on: who is leader, who holds a lock, which members are alive, the current shard map or config. Avoid them for application data, queues, high write rates or large values; that belongs in a database, Kafka or object storage. In an interview, you rarely add one by name; you say the system uses a consensus-backed store for election and config.

Use it when / avoid it when
  • USE when
  • leader election, locks
  • membership, discovery
  • config every node must see
  • consistently
  • AVOID when
  • application data, queues,
  • high write rates, big blobs
  • → databases, Kafka, S3

Numbers

Numbers to have ready. Three or five members, surviving one or two failures. Write latency in one data centre is about one to ten milliseconds, set mostly by disk flush time. Throughput is roughly a thousand to thirty thousand writes a second depending on hardware and value size. etcd caps requests at one and a half megabytes and its database at two gigabytes by default. Session and lease timeouts are usually five to thirty seconds.

Numbers worth knowing
ensemble size        3 or 5
tolerates            1 or 2 failures
write latency        ~1-10 ms (same DC)
writes per second    ~1k - 30k
etcd request max     1.5 MB
etcd db quota        2 GB default
ZK znode max         ~1 MB
session / lease      ~5-30 s typical

Say it like this

How to say it. Only one scheduler instance may run the job at a time. Each instance campaigns for leadership in etcd with a ten second lease; the winner runs, the rest watch the key. Every write the leader makes carries the revision as a fencing token, so a paused former leader cannot corrupt anything. etcd runs as three members across zones, separate from the application database.

In the interview
  • "One scheduler instance must
  • run the job at a time. Each
  • instance campaigns in etcd
  • with a 10 s lease; the winner
  • runs, the rest watch. Each
  • job carries the lease
  • revision as a fencing token,
  • so a stale leader’s writes
  • are rejected."

Interview questions

What are ZooKeeper and etcd used for?

Storing small pieces of coordination state that every node must agree on: leader election, distributed locks, cluster membership, service discovery and configuration. Kubernetes keeps all of its cluster state in etcd.

Why do ensembles have 3 or 5 members?

Writes need a majority, so 3 members tolerate one failure and 5 tolerate two; an even number adds no fault tolerance, and larger ensembles make every write wait for more acknowledgements.

How do you implement leader election with ZooKeeper?

Each candidate creates an ephemeral sequential znode under an election path; the one with the lowest sequence number is leader, and each other candidate watches the node just before its own. When the leader’s session ends, its znode disappears and the next candidate takes over.

What is an ephemeral znode or an etcd lease?

Data tied to a client’s liveness. An ephemeral znode is deleted when its session times out, and keys attached to an etcd lease are deleted when the lease is not kept alive, so crashed clients release their locks and registrations automatically.

What is a fencing token and why is it needed?

An increasing number issued with each lock grant, such as the ZooKeeper zxid or etcd revision. Because a paused client may still believe it holds an expired lock, the protected resource rejects writes with a token lower than one it has already seen.

Are reads from ZooKeeper always up to date?

No. A ZooKeeper member serves reads locally and may lag the leader; call sync before a read when you need the latest value. etcd reads are linearizable by default, with optional faster serializable reads.

Why not store application data in etcd?

It is designed for small values and modest write rates: requests are capped around 1.5 MB, the default database quota is 2 GB, and every write requires a consensus round with disk flushes, so throughput is far below a real database.

What happens when the majority of members is lost?

The remaining minority cannot commit writes or elect a leader, so the service stops accepting changes until a majority is restored. This is the consistency-over-availability choice in CAP terms.