Start practising

SYSTEM DESIGN · CONCEPTS · RUNNING IT IN PRODUCTION · EPISODE 35

Availability, SLAs and the Nines

What availability means, the nines and how much downtime each allows, SLI vs SLO vs SLA and error budgets, how serial and redundant components combine, removing single points of failure, active-passive vs active-active, multi-zone and multi-region designs, and how to talk about availability in an interview.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
1 / 10

Press play to hear the narration, or step through at your own pace. Every picture here is the same one used in the YouTube video.

Notes

The nines

Availability is the share of time the system works. Ninety-nine percent sounds great, but it's three and a half days of downtime a year. Three nines is under nine hours. Four nines is fifty-three minutes. Five nines is five minutes a year. Each extra nine costs roughly ten times more effort.

Downtime per year
99%      (two nines)  3.65 days
99.9%    (three)      8.8 hours
99.95%                4.4 hours
99.99%   (four)       53 minutes
99.999%  (five)       5 minutes

each extra nine costs roughly
10× more effort and money

SLOs

Three terms. The SLI is what you measure, like the percentage of requests that succeed in under three hundred milliseconds. The SLO is your internal target, say ninety-nine point nine percent per month. The SLA is the contract with customers, usually looser, with refunds. The gap to a hundred percent is your error budget: spend it on releases, and when it's gone, slow down and fix reliability.

SLI, SLO, SLA
  • SLI: what you measure
  • % of requests OK in < 300 ms
  • SLO: your internal target
  • 99.9% over 30 days
  • SLA: the promise to customers,
  • with refunds if broken (99.5%)
  • error budget = 100% − SLO
  • 0.1% ≈ 43 minutes a month
  • budget left → ship faster
  • budget gone → freeze, fix

Serial

Components in a chain multiply. If the load balancer, the app and the database are each up ninety-nine point nine percent, a request that needs all three succeeds only ninety-nine point seven percent of the time. Every dependency on the critical path lowers availability, and any single box with no backup is a single point of failure.

In series
  • request needs LB AND app AND DB
  • 0.999 × 0.999 × 0.999
  • = 99.7%
  • every dependency you add
  • to the critical path lowers
  • availability
  • one box with no backup =
  • single point of failure

Redundancy

Redundancy works the other way. Two app servers that are each up ninety-nine percent are both down only one percent of one percent of the time, so together they're ninety-nine point nine nine. But only with health checks, automatic failover, and failures that are truly independent: different racks, different power, different availability zones.

In parallel
  • two app servers, each 99%:
  • both down = 1% × 1% = 0.01%
  • → 99.99% available
  • needs: health checks,
  • automatic failover,
  • independent failures
  • (different zones, racks, power)

Failover

When zone A fails, health checks stop sending it traffic, and the database standby in zone B is promoted. That's active-passive: a standby waits, then takes over. Active-active means every copy serves traffic all the time, so losing one just shifts load, but writes get harder. Either way, provision N plus one, so you survive losing one.

Active-passive vs active-active
  • active-passive: standby waits,
  • promoted on failure
  • simple; failover takes time
  • active-active: all copies serve
  • traffic all the time
  • no idle capacity, instant
  • shift; harder for writes
  • run N+1: survive losing one

Real outages

Here's the uncomfortable truth: most outages are caused by changes, like bad deploys, config pushes or expired certificates, not by hardware. So availability also means canary releases and fast rollbacks, rate limits and load shedding, timeouts and circuit breakers, and graceful degradation: hide recommendations, but keep checkout working.

Beyond hardware
  • most outages are changes:
  • bad deploys, config pushes,
  • expired certificates
  • → canary releases, fast rollback
  • → rate limits, load shedding
  • → timeouts + circuit breakers
  • → graceful degradation:
  • hide recommendations,
  • keep checkout working

Regions

Spreading across availability zones in one region is cheap and standard; it survives a data centre failure. Multiple regions survive a whole region going down, but you pay for replication lag, global load balancing and regular failover drills. How far to go depends on what an hour of downtime costs the business.

Multi-region
  • multi-zone: survives a data
  • centre failure; cheap, standard
  • multi-region: survives a whole
  • region; costly and complex
  • data replication lag
  • DNS / global load balancing
  • failover drills
  • choose by the cost of downtime

Say it like this

In an interview: state the target, say four nines, about fifty-three minutes a year. Walk your diagram and remove every single point of failure: redundant load balancers, N plus one app servers across three zones, a database standby with automatic failover. Then mention graceful degradation, canary deploys and fast rollback.

In the interview
  • state the target: 99.99%
  • = ~53 min/year
  • no single points of failure:
  • LB pair, N+1 app servers
  • across 3 zones
  • DB primary + standby,
  • automatic failover
  • degrade gracefully
  • canary deploys, quick rollback

Interview questions

How much downtime does 99.9% and 99.99% availability allow?

99.9% allows about 8.8 hours per year (about 43 minutes per month). 99.99% allows about 53 minutes per year (about 4.3 minutes per month).

What is the difference between an SLI, an SLO and an SLA?

An SLI is a measured indicator (for example the fraction of successful requests under 300 ms). An SLO is the internal target for it. An SLA is a contractual commitment to customers, usually looser, with penalties if broken.

What is an error budget?

The allowed unreliability implied by the SLO (100% minus the SLO). While budget remains, teams can ship faster; when it is exhausted, they prioritise reliability work and slow releases.

How do serial and parallel components affect availability?

Serial components multiply availabilities, lowering the total (three 99.9% parts give about 99.7%). Independent redundant components multiply failure probabilities, raising it (two 99% parts give 99.99%).

What is a single point of failure and how do you remove it?

A component whose failure stops the whole system. Remove it with redundancy (multiple instances behind health-checked load balancing, replicas with automatic failover) placed in independent failure domains.

Active-passive or active-active?

Active-passive keeps a standby that takes over on failure: simpler but with failover delay and idle capacity. Active-active serves traffic from all copies: no idle capacity and faster recovery, but more complex, especially for writes.

What causes most outages in practice?

Changes: deployments, configuration updates, schema migrations and expiring certificates. Mitigate with canary releases, feature flags, automated rollback and good monitoring.

What is graceful degradation?

Keeping core functionality working when parts fail, by disabling or simplifying non-essential features (recommendations, personalisation) instead of failing completely.