Start practising

SYSTEM DESIGN · PATTERNS · 5 OF 7

Multi-Step Processes: Sagas, State Machines, Outbox and Workflow Engines

How to run a business process that spans several services and can fail halfway, like reserve stock, charge the card, ship and notify. Why one synchronous chain leaves money and stock in limbo, why two-phase commit rarely fits, and the standard moves: a persisted state machine, idempotent retried steps, sagas with compensations, the outbox, and durable workflow engines.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
1 / 15

Press play to hear the narration, or step through at your own pace.

Notes

The signals

The signal is a flow that crosses several services or outside companies, where stopping halfway is unacceptable. Checkout, hotel booking, driver payouts, user onboarding with identity checks. Some steps take minutes or wait for a human approval. When an interviewer asks what happens if payment fails after you reserved the stock, you are in this pattern.

Signals in the question
  • checkout, booking, payouts
  • onboarding, KYC, approvals
  • ride: match, pickup, pay
  • steps owned by different
  • services or companies
  • some steps take minutes or
  • wait for a human
  • "what if payment fails
  • after we reserved stock?"

The naive design

The naive design is one request handler that calls inventory, then payments, then shipping. Now the server crashes right after the charge. The card is charged, stock is reserved forever, nothing ships, and nobody recorded how far it got. If the client retries from the top, the customer is charged twice. Every interviewer will push on exactly this moment.

Why a call chain breaks
  • reserve ok, charge ok,
  • then the server dies
  • the card is charged
  • stock is held forever
  • no shipment, no record of
  • how far we got
  • retry from the top? charges
  • the customer twice

Why not 2PC

Why not a distributed transaction? Two-phase commit asks every participant to prepare, then commit. If the coordinator dies after prepare, everyone waits holding locks. Every participant must support the protocol, and a card network or a courier API never will. So across services we accept that each step commits on its own, and design for recovering forward or undoing. Concepts episode thirty-three covers this.

Why not two-phase commit?
  • 2PC: a coordinator asks all
  • to prepare, then commit
  • blocks if the coordinator
  • dies after prepare
  • every service must support
  • it (card networks do not)
  • locks held across the network
  • fine inside one database
  • rare across services

Move 1: state machine

Move one: make the process a state machine and save its state after every step. The order goes from created to reserved to paid to shipped to done, or into compensating and cancelled. Each transition is a row update. When a server restarts, it loads every order not in a final state and continues from the recorded step. Now a crash pauses the process instead of losing it.

A persisted state machine
CREATED → RESERVED → PAID
  → SHIPPED → DONE
any → COMPENSATING → CANCELLED

orders(id, status, step,
  attempts, updated_at)

after each step: UPDATE status
on restart: load non-final
  orders and continue
from the recorded step

Move 2: idempotent steps

Move two: every step must be safe to retry. When a call times out, you do not know whether it happened, so you will call again. Send an idempotency key, like order eighty-eight pay, and have the payment service store it and return the original result on any repeat. Retry with exponential backoff and jitter. Concepts episode twenty-nine covers idempotency.

Idempotent, retried steps
  • we cannot know if a timed-
  • out call succeeded
  • so every step must be safe
  • to repeat
  • send key = order-88-pay
  • payments stores the key and
  • returns the first result
  • retry with backoff and jitter
  • (Concepts episode 29)

Move 3: saga

Move three: the saga. Each step has a compensating action that semantically undoes it: release the stock, refund the charge, cancel the courier. If the card is declined after stock was reserved, run the compensations for the completed steps in reverse order. It is not a rollback; the charge really happened and the refund really happens. Put irreversible steps like emails last, and make compensations idempotent too.

Saga with compensations
step          compensation
reserve       release stock
charge        refund
book courier  cancel booking
email         (none, last)

charge fails → release stock
ship fails → refund, release

compensations must also be
idempotent and retried

Orchestrate or react

There are two ways to run a saga. Orchestration: one coordinator calls each step and owns the state, so the whole flow is visible in one place and timeouts are easy. Choreography: each service listens for the previous event, like stock reserved, and emits its own. It is loosely coupled, but past four or so steps nobody can see the whole flow. In interviews, orchestration is usually the clearer answer.

Two ways to run a saga
  • orchestration: one service
  • calls each step, owns state
  • + flow is visible in one place
  • + easy to add timeouts
  • choreography: each service
  • reacts to the last event
  • + loose coupling
  • - flow scattered, hard to
  • debug past ~4 steps

Move 4: outbox

Move four: the outbox. A service often needs to update its database and publish an event. Do those separately and a crash between them either loses the event or announces something that never committed. Instead, write the event into an outbox table in the same local transaction. A relay process reads the outbox, or the database change log, publishes, and marks it sent. Consumers may see it twice, so they dedupe.

Transactional outbox
BEGIN;
  UPDATE payments SET ...;
  INSERT INTO outbox
    (event, payload);
COMMIT;

relay reads outbox (or the
  DB change log) → publishes
  → marks sent

no lost or phantom events

Move 5: workflow engine

Move five: a workflow engine like Temporal or AWS Step Functions. You write the steps as ordinary code, and the engine records each step's result durably. If a worker crashes, another one replays the history and carries on from the same point. You get retries, timeouts, durable timers like wait three days, and signals for human approval for free. The cost is one more system to operate.

Move 5: workflow engine
  • Temporal, AWS Step Functions,
  • Cadence, Airflow (batch)
  • you write the steps as code;
  • the engine persists each
  • result and replays on crash
  • durable timers: "wait 3 days"
  • built-in retries, timeouts,
  • human-approval signals
  • cost: another system to run

Failure modes

Some processes get stuck no matter what. Give every step a deadline. Run a sweeper that finds orders sitting in one state for too long and either retries or compensates. When retries are exhausted, move the item to a dead-letter queue and alert a human, with a small console to repair it. For money, reconcile every night against the provider's report, because that report is the final truth.

Stuck and slow processes
  • every step has a deadline
  • sweeper: find orders in one
  • state longer than expected
  • retries exhausted → dead-
  • letter queue + alert
  • manual repair console for
  • the last 0.01%
  • reconcile nightly with the
  • payment provider report

Trade-offs

The trade-offs. You give up atomicity: there are visible in-between states, like paid but not yet shipped, so the UI must show them honestly. Compensations are not free; refunds cost fees and an email cannot be unsent. An orchestrator is clear but central. A workflow engine saves code but adds operations. Whatever you choose, tag every log and trace with the order ID so you can follow one process end to end.

Trade-offs
  • eventual consistency: the
  • order is briefly PAID but
  • not SHIPPED
  • compensations are not free:
  • refunds cost fees, emails
  • cannot be unsent
  • orchestrator: clear, but a
  • central dependency
  • engine: less code, more ops
  • trace with one order_id

Where it shows up

Where you will use this: e-commerce checkout is the textbook saga. The payment system charges, writes the ledger and pays out. Movie booking holds, pays and confirms or releases. Uber and food delivery move an order through matching, pickup, delivery and payment. And a CI pipeline builds, tests, deploys and rolls back, which is a multi-step workflow with compensations.

Problems that use it
  • E-commerce: checkout saga
  • Payment system: charge,
  • ledger, payout
  • Movie booking: hold, pay,
  • confirm or release
  • Uber, food delivery: match,
  • pickup, deliver, pay
  • CI/CD pipeline: build, test,
  • deploy, roll back

Say it like this

The phrasing. Checkout is a saga run by an orchestrator. The order's state is persisted after each step, so a crash only pauses it. Every call carries an idempotency key and is retried with backoff. If payment fails, we compensate by releasing the stock. Events leave through an outbox. A sweeper finds stuck orders, and anything unrecoverable goes to a dead-letter queue for a human.

In the interview
  • "Checkout is a saga run by an
  • orchestrator. Order state is
  • persisted after each step.
  • Every call carries an
  • idempotency key and is
  • retried. If payment fails,
  • we release the stock. Events
  • leave via an outbox. A
  • sweeper handles stuck ones."

Interview questions

What is a saga?

A long-running business transaction split into local steps, each committed independently and each paired with a compensating action. If a later step fails, the completed steps are compensated in reverse order instead of being rolled back.

Why not use two-phase commit across microservices?

It blocks participants holding locks if the coordinator fails, adds network round trips to every transaction, and requires every participant to support it, which external systems such as payment providers do not.

Orchestration or choreography?

Orchestration puts the flow in one coordinator, which makes it easy to see, change and time out. Choreography lets services react to each other’s events, which is loosely coupled but hard to follow beyond a few steps; most interview answers favour orchestration for core flows.

What problem does the transactional outbox solve?

It removes the dual-write problem: the state change and the event are written in one local database transaction, and a separate relay publishes the event afterwards, so an event is never lost or published for a change that did not commit.

How do you make retries safe in a multi-step flow?

Give every step a stable idempotency key derived from the process, such as order ID plus step name, and have the receiving service store the key with its result so repeated calls return the same result without repeating the side effect.

What do you do with a process that is stuck?

Detect it with per-step deadlines and a sweeper that finds records in one state for too long, retry or compensate automatically, and after a retry limit move it to a dead-letter queue with an alert and a manual repair tool.

When would you use a workflow engine such as Temporal?

When processes are long-running, have many steps, timers or human approvals, and you want durable execution, retries and visibility without hand-building state machines and sweepers in every service.