Start practising

SYSTEM DESIGN · CONCEPTS · DISTRIBUTED SYSTEMS · EPISODE 29

Idempotency and Retries

Why retries cause double charges and duplicate orders, what idempotent means, which HTTP methods are idempotent, idempotency keys stored with the result, making queue consumers safe with dedup tables and upserts, and retrying well with timeouts, exponential backoff and jitter.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
1 / 11

Press play to hear the narration, or step through at your own pace. Every picture here is the same one used in the YouTube video.

Notes

The problem

Here's the bug. The app sends 'pay nine ninety-nine'. The server charges the card, but the response is lost on a bad mobile connection. From the app's side, a lost request and a lost response look identical: a timeout. So it retries. And the customer is charged twice.

The lost response
  • request reached the server
  • card was charged
  • response timed out on the
  • way back (bad mobile signal)
  • the app cannot tell:
  • did it fail, or did only
  • the answer get lost?
  • so it retries → charged twice

Definition

An operation is idempotent if doing it twice has the same effect as doing it once. 'Set balance to five hundred' is idempotent. 'Subtract one hundred' is not. In HTTP, GET, PUT and DELETE are meant to be idempotent; POST usually is not. Payments, orders and messages are almost always POSTs, so we have to make them idempotent ourselves.

Idempotent
  • doing it twice has the same
  • effect as doing it once
  • idempotent:
  • SET balance = 500
  • DELETE order 42
  • PUT /users/7 {…}
  • not idempotent:
  • balance = balance − 100
  • INSERT INTO orders …
  • POST /pay

Idempotency key

The standard fix is an idempotency key. When the user taps Pay, the app generates a unique id once, and sends it with the request and with every retry of that same action. The server stores the key together with the result. Stripe's API works exactly like this.

Idempotency key
POST /pay
Idempotency-Key: 7f3a9c1e
{ amount: 999 }

client creates the key ONCE
per user action and reuses it
on every retry

Stripe, PayPal, Adyen APIs
all work this way

The retry

On the retry, the server looks up the key, finds 'already paid', and returns the stored response without charging again. Insert the key under a unique constraint before doing the work, so two retries arriving at the same moment can't both proceed. Keep keys for a day or so, then expire them.

Server logic
row = SELECT … WHERE key = ?
if row: return row.response

INSERT key (UNIQUE) status=busy
charge card
UPDATE key SET response = …
return response

UNIQUE constraint stops two
concurrent retries

Consumers

The same idea fixes queue consumers from episode twenty-eight, which receive duplicates. Either record each processed message id in a table with a unique constraint and skip repeats, ideally in the same transaction as the work, or design the write to be naturally idempotent: an upsert of 'order forty-two is shipped' can run twice harmlessly.

Idempotent consumers
queues deliver at-least-once
(episode 28)

dedupe by message id:
INSERT INTO processed(msg_id)
  → conflict? skip

or write with natural keys:
UPSERT order_status
WHERE order_id = 42

Retries

Now retry well. Put a timeout on every call. Retry only failures that might succeed later: timeouts, 503s, resets, never a 400. Wait longer each time, exponential backoff, and add random jitter so thousands of clients don't retry in the same millisecond. And cap the attempts.

Retry well
  • timeout every call
  • retry only safe failures:
  • timeouts, 503, connection reset
  • NOT 400 bad request
  • exponential backoff:
  • 100 ms, 200, 400, 800…
  • + random jitter
  • cap the attempts

Retry storms

Retries can cause outages. A service slows down, every client retries three times, and now the sick service gets four times the traffic. Limit retries with a budget, use circuit breakers that stop calling a failing service for a while, and retry at one layer only, not in the app, the SDK and the load balancer all at once.

Retry storms
  • service slows down
  • → every client retries 3×
  • → 4× the traffic on a
  • service that is already sick
  • fixes:
  • retry budgets (≤ 10% extra)
  • circuit breakers: stop calling
  • a failing service for a while
  • retry at ONE layer only

Decision

Where do you need this? Payments, refunds and transfers. Order creation and bookings. Sending emails, texts and push notifications. Webhook handlers, because senders retry for days. And every queue consumer. The rule: if doing it twice would cost money or trust, make it idempotent.

Where you need it
  • payments, refunds, transfers
  • order creation, bookings
  • sending email / SMS / push
  • webhook handlers
  • (senders retry for days)
  • queue consumers
  • rule: if a duplicate costs
  • money or trust, make it
  • idempotent

Say it like this

The interview answer. The client generates an idempotency key per user action and reuses it on retries. The server stores the key and the result under a unique constraint and returns the stored result on repeats. Consumers dedupe by message id. Retries use timeouts, exponential backoff with jitter, and a cap.

In the interview
  • "What if the client retries?"
  • client-generated idempotency
  • key per action
  • server stores key + result
  • with a unique constraint
  • return stored result on repeat
  • consumers dedupe by msg id
  • retries: backoff + jitter + cap

Interview questions

What does idempotent mean?

Performing the operation multiple times has the same effect as performing it once. Setting a value is idempotent; incrementing a counter or inserting a new row is not.

Why do retries create duplicates?

A client that times out cannot distinguish a request that never reached the server from one that succeeded but whose response was lost, so it retries, and a non-idempotent operation is executed again.

Which HTTP methods are idempotent?

GET, HEAD, PUT, DELETE and OPTIONS are defined as idempotent. POST and PATCH are not guaranteed to be, so APIs make POST endpoints idempotent with idempotency keys.

How do idempotency keys work?

The client generates a unique key per logical action and sends it with every attempt. The server records the key and the result, using a unique constraint, and on a repeated key returns the stored result instead of executing again.

How do you make a queue consumer idempotent?

Record processed message ids in a table with a unique constraint, ideally in the same transaction as the side effect, and skip duplicates; or use naturally idempotent writes such as upserts keyed by entity id.

What are exponential backoff and jitter?

Waiting progressively longer between retries (for example 100, 200, 400 ms) and adding randomness to each delay so many clients do not retry in synchronised waves.

What is a retry storm and how do you prevent it?

Retries multiply load on a service that is already struggling, turning a slowdown into an outage. Prevent it with retry budgets, circuit breakers, backoff with jitter, and retrying at only one layer.

Which failures should not be retried?

Client errors such as 400 Bad Request, 401 or 403, and validation failures, because they will fail again. Retry timeouts, connection resets and 503 or 429 responses (respecting Retry-After).