Start practising

SYSTEM DESIGN · CONCEPTS · DISTRIBUTED SYSTEMS · EPISODE 33

Distributed Transactions: 2PC, Sagas, Outbox

Why one transaction cannot span several services and databases, two-phase commit and its blocking problem, sagas with compensating actions, orchestration vs choreography, the dual-write problem and the transactional outbox, and how to answer "what if payment fails after the order is created?"

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
1 / 10

Press play to hear the narration, or step through at your own pace. Every picture here is the same one used in the YouTube video.

Notes

The problem

Inside one database, a transaction is all or nothing. But here the stock was reserved in one database and the payment failed in another, and no single commit covers both. We're left half done: stock held for an order that will never be paid. We need another way to keep services consistent.

Half done
  • one DB: BEGIN … COMMIT
  • all or nothing (episode 18)
  • three services, three DBs:
  • stock reserved ✓
  • payment failed ✗
  • order? half created
  • no single BEGIN/COMMIT
  • spans them

Two-phase commit

The classic answer is two-phase commit. Phase one: a coordinator asks every participant to prepare: do the work, lock the rows, make it durable, and vote. Phase two: if everyone voted yes, the coordinator says commit; if anyone said no, everyone aborts. It's truly atomic across databases.

Two-phase commit
  • phase 1, prepare: each DB does
  • the work, locks rows, writes it
  • durably, votes yes or no
  • phase 2: all yes → commit
  • any no → everyone aborts
  • atomic across databases

2PC problems

The problem is blocking. If the coordinator crashes after participants have prepared, they sit holding locks, unable to commit or abort on their own. It's also slow, with locks held across several round trips, and it needs every participant available. So it's used inside some databases, but rarely across microservices.

Why services avoid 2PC
  • blocking: coordinator crashes
  • between phases → participants
  • hold locks, cannot decide
  • slow: several round trips,
  • locks held the whole time
  • needs every participant up
  • used inside some databases,
  • rarely across microservices

Sagas

The common answer is a saga. Break the work into local transactions, each committing in its own service, and give each a compensating action: cancel the order, release the stock, refund the card. If a step fails, run the compensations for the steps already done, in reverse. Here, payment failed, so release the stock and cancel the order.

Saga
  • step compensation
  • create order cancel order
  • reserve stock release stock
  • charge card refund
  • ship (none: last)
  • each step commits locally
  • failure → run compensations
  • backwards

Saga details

Sagas can be orchestrated, with a coordinator that stores the saga's state and runs each step, or choreographed, with services reacting to each other's events. Remember they aren't isolated: others can see the order as 'pending' in between. And compensations must be idempotent and retryable, because they'll be retried.

Saga details
  • orchestrated: a coordinator
  • service runs the steps and
  • stores saga state
  • choreographed: each service
  • reacts to the previous event
  • not isolated: others can see
  • "pending" in between
  • compensations must be
  • idempotent and retryable

Dual writes

There's a sneakier bug inside every saga step: the dual write. A service saves the order to its database, then publishes an event. If it crashes between the two, the order exists but nobody hears about it. Swap the order and you can announce an order that was never saved.

The dual-write problem
db.save(order)        ✓
kafka.publish(event)  ✗ crash

→ order exists, nobody hears
  about it

or the reverse: event sent,
DB write rolled back

Outbox

The fix is the transactional outbox. In the same local transaction that saves the order, insert the event into an outbox table. Both commit or neither does. A separate relay reads the outbox, often via change data capture, and publishes to Kafka. Delivery is at least once, so consumers stay idempotent.

Transactional outbox
BEGIN
  INSERT INTO orders …
  INSERT INTO outbox
    (event = OrderCreated)
COMMIT       ← one local tx

relay reads outbox → Kafka
(polling or change data capture)
at-least-once → consumers
must be idempotent (ep. 29)

Say it like this

In an interview, when asked what happens if payment fails: use a saga, orchestrated by an order workflow, where each step is a local transaction with a compensation like release stock, refund or cancel. Publish events with the outbox pattern, keep every step idempotent, and show users a pending status. Avoid two-phase commit across services.

In the interview
  • "What if payment fails?"
  • saga, orchestrated by an order
  • workflow; each step local
  • compensations: release, refund,
  • cancel; idempotent + retried
  • outbox for reliable events
  • show "pending" status to users
  • avoid 2PC across services

Interview questions

Why can you not use a normal database transaction across microservices?

Each service owns a separate database, and a local transaction only covers one database. There is no shared commit point, so failures can leave some services updated and others not.

How does two-phase commit work?

A coordinator asks all participants to prepare (do the work durably, hold locks and vote). If all vote yes it tells them to commit; if any votes no or times out, all abort.

What are the drawbacks of two-phase commit?

It is blocking: if the coordinator fails after prepare, participants hold locks and cannot decide. It adds latency, reduces availability because all participants must be up, and couples services.

What is a saga?

A sequence of local transactions, one per service, where each step has a compensating action. If a later step fails, compensations undo the completed steps in reverse order, giving eventual consistency.

Orchestrated or choreographed saga?

Orchestration uses a central coordinator that tracks state and calls each step, which is easier to understand and monitor. Choreography lets services react to events with no coordinator, which is looser but harder to follow as flows grow.

What is the dual-write problem?

Writing to a database and publishing a message as two separate operations. A crash between them leaves the two out of sync, such as a saved order with no event, or an event for an order that was rolled back.

How does the transactional outbox pattern work?

The service writes the business change and an event row to an outbox table in one local transaction. A relay (polling or change data capture) later publishes outbox rows to the message broker with at-least-once delivery.

Why must saga steps and compensations be idempotent?

Messages and commands are retried after timeouts and crashes, so a step may run more than once; idempotency ensures repeats do not double-charge or double-release.