Notes
Requirements
The requirements. The breaker wraps any call and takes a fallback. It tracks the last N outcomes and trips when the failure rate reaches a threshold, but only after a minimum number of calls, so two early errors do not trip it. Calls slower than a limit count as failures. While open, calls fail fast to the fallback. After a cool-down, a few trial calls decide: all succeed and it closes, one fails and it reopens.
Pin the numbers. A count-based window of the last ten calls, tripping at fifty percent once at least five calls are in. Open for thirty seconds, then half-open with two trial calls; anyone else arriving during the trial fails fast. A failure is an exception or a call slower than two seconds. And there is one breaker per dependency, so a broken payments service never blocks calls to the inventory service.
- wrap any call: call(action, fb)
- track the last N outcomes
- trip when failure rate >= X%
- but only after M calls
- slow calls count as failures
- open: fail fast to a fallback
- after a cool-down: a few trials
- all pass: close; one fails: open
- notify on every state change
- safe from many threads
- window by count or by time?
- count: last 10 calls
- trip threshold?
- 50% of at least 5 calls
- how long open?
- 30 s, then half-open
- how many trials?
- 2; extra callers fail fast
- what is a failure?
- exception or slower than 2 s
- one breaker per dependency
Entities
The entities. The breaker holds its current state and a sliding window of recent outcomes. A config record holds the numbers: window size, minimum calls, threshold, slow-call limit, cool-down and trial count. A clock is injected so tests can jump thirty seconds instantly. And each call brings its own fallback, a function from the error to a default answer.
State machine
The heart is a state machine with three states and four transitions. Closed goes to open when the failure rate crosses the threshold. Open goes to half-open when the cool-down has passed. Half-open goes back to closed when every trial succeeds, and back to open on the first failed trial. Entering closed resets the window, so old failures cannot trip it again a moment later. Draw this diagram first in the interview; everything else is bookkeeping.
- CLOSED -> OPEN
- failure rate >= 50% of >= 5
- OPEN -> HALF_OPEN
- 30 s have passed
- HALF_OPEN -> CLOSED
- 2 trials succeeded
- HALF_OPEN -> OPEN
- any trial failed
- closed: window reset
Class diagram
In code, the State pattern can be a full class per state, but with three states and two events an enum with an exhaustive switch is clearer, and the compiler checks every state is handled. There are only two events: may I call, and here is the result. The config is a record that validates itself, so a window smaller than the minimum call count is rejected at construction.
enum State { CLOSED, OPEN, HALF_OPEN }
record Config(int windowSize,
int minCalls, int failureRatePercent,
long slowCallMillis, long openMillis,
int halfOpenTrials)
switch (state) {
case CLOSED -> true;
case OPEN -> false;
case HALF_OPEN -> trialsStarted++ < 2
} Sliding window
The sliding window is a ring buffer of the last ten outcomes with a running failure count. Recording overwrites the oldest slot; if the outcome being dropped was a failure, the count goes down first. So the failure rate is one division, with no scan, on every call. If the interviewer prefers a time-based window, keep one bucket per second with counts, and drop buckets older than the window.
boolean[] failed; int next, size, fails record(f): if full and failed[next]: fails-- else if not full: size++ failed[next] = f; if (f) fails++ next = (next + 1) % capacity rate = fails * 100 / size time-based: buckets per second
Flow: closed to open
The closed flow. Ask for permission, which in closed is always yes. Run the action outside any lock, time it, and record the outcome. In the demo the payments service starts returning errors. After one success and four failures there are five calls in the window at eighty percent failure, which crosses fifty, so the breaker opens. The caller still gets an answer, the fallback, which queues the payment for later.
if (!tryAcquirePermission())
return fallback(new NotPermitted())
start = now
try {
r = action.get() // outside the lock
onResult(now - start > slowMillis)
return r
} catch (RuntimeException e) {
onResult(true); return fallback(e)
} Flow: open
While open, the remote service is not called at all. The demo checks that the remote call counter does not move. The fallback is a per-call strategy: queue the payment, serve a cached value, return an empty list of recommendations, or hide a widget. The refusal arrives as its own exception type, so the fallback can tell not tried from tried and failed, which matters for logging and for retries.
- queue it for later (payments)
- serve the cached value
- return a safe default
- (empty recommendations)
- degrade: hide the widget
- or fail fast with a clear error
- refusal is an exception type,
- so the fallback can tell
- "not tried" from "tried, failed"
Flow: half-open
Half-open. There is no timer thread: whenever someone asks for permission, the breaker checks whether the cool-down has passed, and if so moves to half-open. That keeps the design simple and testable. In the demo the first trial fails, so it reopens for another thirty seconds. Next time two trials succeed and the breaker closes with a fresh window.
void maybeHalfOpen() {
if (state == OPEN
&& now - openedAt >= openMillis)
transition(HALF_OPEN)
}
called from tryAcquirePermission()
and state(): no timer thread
HALF_OPEN: trial fails -> OPEN
2 trials succeed -> CLOSED Timeouts
Timeouts. A dependency that answers in three seconds hurts as much as one that errors, because your threads are stuck, so calls slower than the limit count as failures; in the demo, five slow calls trip the breaker. But a call that hangs forever never reports at all, so also bound the wait itself, with a client timeout on the HTTP or database driver, or a future with a timed get. Keep that timeout shorter than the caller's own deadline.
- slow: took > 2 s, count as failed
- 5 slow calls -> OPEN
- but a call that never returns
- is never recorded at all
- so also bound the wait:
- client timeout (HTTP, JDBC)
- or future.get(2, SECONDS)
- then cancel the task
- timeout below the caller budget
Retry with jitter
Retries sit outside the breaker. Each failed attempt waits longer, a hundred, two hundred, four hundred milliseconds, capped. Full jitter picks a random wait between zero and that ceiling, so a thousand clients that failed together do not all retry in the same millisecond. Never retry a call the breaker refused, because that defeats failing fast, and only retry calls that are safe to repeat, or that carry an idempotency key.
for attempt in 1..maxAttempts:
try { return action.get() }
catch (NotPermitted e) { throw e }
catch (RuntimeException e) {
ceil = min(cap, base << (attempt-1))
sleep(random * ceil) // full jitter
}
slept [73, 149] ms: under 100, 200
retry only idempotent calls Concurrency
Concurrency. The two decisions, may I call and record this result, are short synchronized methods, constant time each. The remote call itself runs with no lock held. In half-open, the breaker counts trials started, not finished, so when eight threads race in at once exactly two are admitted and six fail fast; the demo checks that. A result that arrives after the breaker already tripped is ignored. At very high request rates, use an atomic state and striped counters instead.
- synchronized: tryAcquirePermission
- and onResult, both O(1)
- the remote call: no lock held
- half-open: count trials started,
- not finished, so 8 racing
- threads admit exactly 2
- late result after a trip: ignore
- hot path at high QPS: an atomic
- state + striped counters
Observer
Patterns. State for the three modes, Proxy because the breaker wraps the real call, Strategy for the fallback, and Observer for transitions: an alerting listener pages someone when payments opens, a dashboard colours it red, a log line records it. Retry, breaker and client stack as decorators around one call. Listeners must be quick, or be handed off to another thread, since they run during the transition.
- State: CLOSED, OPEN, HALF_OPEN
- Proxy: call() wraps the action
- Strategy: the fallback per call
- Observer: StateListener
- Decorator: Retry around breaker
- around the client
Extensibility
Extensions. Swap the count window for per-second buckets over a minute. Add a predicate for which errors count, because a not-found answer is not an outage. Add a bulkhead, a semaphore that limits concurrent calls to one dependency. Grow the cool-down after repeated failed trials. Keep breakers in a registry keyed by dependency name, and give operators a manual force-open switch for incidents.
- time-based window: per-second
- buckets over the last 60 s
- ignore some errors: 404 is not
- an outage (a predicate)
- bulkhead: max concurrent calls
- growing cool-down: 30 s, 60 s..
- registry: one breaker per
- dependency, made on demand
- manual force-open for incidents
Testing
The full code is below the player. The demo trips the breaker after four failures, proves the open breaker never calls the service, fails one trial and reopens, then closes after two good trials. Slow calls trip it again. Then eight threads race into half-open and exactly two are admitted, using latches so the test never depends on timing. Finally a seeded retry succeeds on the third attempt with jittered waits under each ceiling.
Call 3: queued for later (503 from payments) CLOSED -> OPEN While open: queued for later (... fast) OPEN -> HALF_OPEN, trial fails HALF_OPEN -> OPEN 2 trials pass: HALF_OPEN -> CLOSED After 5 slow calls: OPEN Half-open race: 2 of 8 admitted Retry: charged on attempt 3, slept [73, 149] ms
Follow-ups
Expect follow-ups. How do timeout, breaker and retry relate? They stack: the timeout bounds each attempt, the breaker decides whether to attempt at all, and retry sits outside both. Should breakers share state across service instances? Usually not; each instance's local view reacts fast and needs no coordination. Why a minimum call count? One failure out of two is noise. And service meshes do the same per host as outlier detection.
- breaker vs retry vs timeout?
- they stack: timeout inside,
- breaker, then retry outside
- shared across instances?
- usually not: local view is
- fast and good enough
- why minimum calls?
- 1 of 2 failing is noise
- where in a service mesh?
- outlier detection per host
Say it like this
The short version. Three states. Closed records each outcome in a ring buffer with a running failure count and trips at fifty percent of at least five calls, counting slow calls as failures. Open fails fast to a fallback without touching the dependency. After a cool-down, a fixed number of trial calls decide whether to close or reopen. Decisions are short synchronized steps, and retry with jittered backoff wraps the outside.
- "Three states. Closed records
- outcomes in a ring buffer with a
- running failure count and trips
- at 50% of at least 5 calls,
- slow calls included. Open fails
- fast to a fallback. After the
- cool-down, two trials decide.
- Retry with jitter goes outside."