Start practising

SYSTEM DESIGN · CONCEPTS · RUNNING IT IN PRODUCTION · EPISODE 40

Failure Handling: Timeouts, Circuit Breakers, Backpressure

How to stop one slow dependency from taking down a whole system: cascading failures, timeouts and deadlines, retries with backoff, circuit breakers and their three states, bulkheads, load shedding and backpressure, fallbacks and graceful degradation, and chaos testing.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
1 / 10

Press play to hear the narration, or step through at your own pace. Every picture here is the same one used in the YouTube video.

Notes

Cascading failure

Watch a cascading failure. The recommendations service gets slow, answering in thirty seconds. The product service's threads each wait for it, until all two hundred are stuck. Now it can't serve any request at all, even ones that never needed recommendations, and its own callers start piling up. A slow dependency is worse than a dead one.

How one slow service kills all
  • recommendations responds in 30 s
  • product service threads wait
  • for it, one by one
  • 200 threads, all stuck waiting
  • → no thread left for ANY request
  • → product page down
  • → callers of product pile up too
  • slow is worse than dead

Timeouts

Rule one: every network call needs a timeout, and many client libraries default to none. Set it a bit above the dependency's p ninety-nine: if it normally answers within a hundred and twenty milliseconds, give up after two hundred. Better still, pass a deadline along, so downstream calls know how much time is left. Retries stay few, with backoff and jitter.

Timeouts and deadlines
  • no timeout = infinite wait
  • (many clients default to none!)
  • set from the dependency's p99:
  • p99 120 ms → timeout ~200 ms
  • deadline propagation:
  • request has 1 s total; pass the
  • remaining time downstream
  • retries: few, backoff + jitter

Circuit breaker

A circuit breaker sits around a dependency. While closed, calls pass and failures are counted. When too many fail, it opens: calls fail instantly without even trying, giving the sick service room to recover and freeing your threads. After a pause it goes half-open and lets a few test calls through. If they succeed, it closes again.

Circuit breaker states
  • CLOSED: calls go through,
  • failures are counted
  • 50% failing → OPEN:
  • fail fast, no calls at all,
  • for 30 seconds
  • HALF-OPEN: let a few test
  • calls through
  • succeed → CLOSED
  • fail → OPEN again

Fallbacks

When a call fails or the breaker is open, return something useful. Recommendations down? Show cached popular items. Reviews slow? Load them after the page. But pricing is essential: without a price you can't sell, so that one fails clearly. Decide in advance which features are optional and what each one falls back to.

Graceful degradation
  • recommendations down →
  • show "popular items" (cached)
  • reviews slow → load them later
  • pricing down → can't sell:
  • this one must fail clearly
  • decide in advance which
  • features are optional and
  • what each falls back to

Bulkheads

Bulkheads come from ships, which are split into compartments so one leak doesn't sink everything. Give each dependency its own thread or connection pool: fifty for pricing, fifty for inventory, twenty for recommendations. Now a slow recommendations service can only exhaust its own twenty threads, and the rest of the page keeps working.

Bulkheads
  • ships have compartments so one
  • leak does not sink the ship
  • separate thread / connection
  • pools per dependency:
  • pricing: 50 threads
  • inventory: 50 threads
  • recommendations: 20 threads
  • slow recs can only exhaust
  • its own 20

Overload

When overloaded, shed load: reject excess requests quickly with a four twenty-nine or five oh three and a retry-after, starting with low-priority work. Inside the system, use backpressure: bounded queues, so when a consumer falls behind, producers are slowed or told no instead of filling memory until everything crashes. Serving eighty percent beats serving nobody.

Load shedding and backpressure
  • overloaded? reject early:
  • 429 / 503 with Retry-After
  • drop low-priority work first
  • backpressure: a slow consumer
  • tells producers to slow down
  • bounded queues, not infinite
  • queue full → producer waits
  • or is told no
  • serving 80% beats serving 0%

Chaos testing

None of this counts until it's tested. Chaos engineering means breaking things on purpose, carefully: kill instances, inject latency, take away a whole zone, and watch whether timeouts, breakers and fallbacks really work. Practise incidents in game days, and load-test above your expected peak.

Test it
  • chaos engineering: kill pods,
  • add latency, drop a zone,
  • on purpose, in a controlled way
  • game days: practise incidents
  • load tests above peak
  • find the cascade before
  • your users do

Say it like this

In an interview, for every dependency ask: what if it's slow or down? Then answer with timeouts based on p ninety-nine and propagated deadlines, a few idempotent retries with backoff and jitter, a circuit breaker per dependency, bulkhead pools, a cached or partial fallback, and load shedding with bounded queues.

In the interview
  • "What if X is slow or down?"
  • timeouts from p99 + deadlines
  • retries: few, backoff, jitter,
  • idempotent
  • circuit breaker per dependency
  • bulkhead pools
  • fallback: cached / partial data
  • load shedding + bounded queues

Interview questions

What is a cascading failure?

A failure that spreads: a slow or failing dependency causes callers to exhaust threads, connections or memory while waiting, so they fail too, and the failure propagates up through the system.

How do you choose a timeout value?

Base it on the dependency's normal latency, typically a little above its p99, and within the caller's own deadline. Propagate the remaining deadline downstream so nested calls do not exceed it.

How does a circuit breaker work?

Closed: calls pass and failures are counted. When the failure rate crosses a threshold it opens and fails fast without calling. After a cool-down it goes half-open and allows trial calls; success closes it, failure reopens it.

What is the bulkhead pattern?

Isolating resources such as thread pools, connection pools or instances per dependency or tenant, so that one misbehaving dependency can only exhaust its own share.

What is load shedding?

Deliberately rejecting some requests when overloaded, quickly and preferably low-priority ones first, so the system keeps serving the rest instead of collapsing.

What is backpressure?

A mechanism for a slow consumer to signal producers to slow down, typically through bounded queues or flow control, preventing unbounded buffering and memory exhaustion.

What is graceful degradation?

Continuing to provide core functionality with reduced features when dependencies fail, for example showing cached popular items instead of personalised recommendations.

Why is a slow dependency often worse than a dead one?

A dead dependency fails fast and frees resources; a slow one ties up threads and connections for the full wait, starving unrelated requests and causing cascading failures.