Design a Payment System: Idempotency, State Machines, Ledgers and Reconciliation

Listen

Clarify scope. The order service asks us to charge a customer's card or wallet, often as an authorization first and a capture when the item ships. We support full and partial refunds, report status to orders and support, and record what we owe sellers. Non-functional needs dominate: money moves exactly once, every amount is auditable for years, the service stays up, and we never touch raw card data. We integrate with providers; we are not building a card network.

Ask scale

The numbers are modest. Five million payments a day is about sixty a second, maybe a thousand at a sale peak. Each payment writes a row, a few attempt rows and a few ledger entries, about three kilobytes, so five and a half terabytes a year, kept for seven to ten years for audits and disputes. The number that really matters is the provider's latency: usually under a few seconds, sometimes thirty, and sometimes no answer at all.

5M payments/day = 60 TPS avg
  sale day peak x15 = about 1,000 TPS
per payment: 1 payment row, 1-3
  attempts, 2-4 ledger entries
  about 3 KB total
5M x 3 KB = 15 GB/day = 5.5 TB/year
keep 7-10 years (audit, disputes)

provider call: 300 ms - 3 s
  sometimes 30 s or no answer

Define API

Every write takes an idempotency key, chosen by the caller and tied to the business action, like order eight eight one, pay. Amounts are integers in the smallest currency unit, so a thousand four hundred and ninety-nine rupees is one four nine nine double zero paise; never use floating point for money. The payment method is a token, never a card number. Create, read, capture and refund, plus a signed webhook endpoint for the provider.

POST /v1/payments
  Idempotency-Key: ord_881_pay
  { order_id, amount: 149900,
    currency: "INR", method: "tok_x" }
  -> 201 { id, status: PROCESSING }
GET  /v1/payments/{id}
POST /v1/payments/{id}/capture
POST /v1/payments/{id}/refunds
  Idempotency-Key, { amount }
POST /webhooks/provider  (signed)

Data model

This is the textbook case for a relational database with ACID transactions. A payments table with a unique idempotency key, a hash of the request, the amount, status and a version. An attempts table, one row per call to a provider, so retries are visible. A ledger of append-only entries where every transaction sums to zero. And an idempotency table holding the saved response for each key. At larger scale, shard by merchant or payment id; at sixty a second, one primary with replicas is plenty.

payments(id, order_id, idem_key UNIQUE,
  request_hash, amount, currency,
  status, provider, version)
attempts(id, payment_id, provider,
  provider_ref, status, error, at)
ledger_entries(id, txn_id, account,
  direction, amount, currency, at)
  append only, sum per txn = 0
idempotency(key, response, expires)
shard by merchant or payment id

High-level design

Trace one payment. The order service calls us with an idempotency key. We first write the payment as processing, with the key, in our own database. Only then does the executor call the provider, sending our attempt id as the provider's own idempotency reference. Writing intent before calling out is the golden rule: if we crash after the call, we still know the call may have happened.

When the provider approves, one database transaction moves the payment to authorized, writes balanced ledger entries, and inserts a payment succeeded event into an outbox that is relayed to the bus for the order service. Later, the provider also sends webhooks about captures, refunds and disputes. The webhook handler verifies the signature, ignores events it has already processed, and applies the rest through the same state machine.

Deep dive: idempotency

Idempotency is the first deep dive. Insert the key with a hash of the request body. If the insert succeeds, process the payment and save the response under the key. If the key exists with the same body, return the saved response, or a 409 if it is still in progress. If it exists with a different body, that is a client bug, so return 422. Then do the same downstream: our attempt id is the provider's idempotency key, so our own retries cannot double charge either. Concepts episode 29 covers this.

INSERT INTO idempotency (key, hash)
  ON CONFLICT DO NOTHING
inserted -> process, save response
conflict, same hash -> return saved
  response (or 409 if in progress)
conflict, other hash -> 422 misuse

downstream: our attempt id becomes
  the provider idempotency key
keep keys at least 24 h-30 days

Deep dive: states

Model the payment as an explicit state machine: created, processing, requires action when the bank asks for a one-time password, authorized or failed, then captured or voided, then partially or fully refunded. Enforce every transition inside the update, setting the new status only where the current status is the expected one. A late webhook or a duplicate retry then cannot move a payment backwards or skip a step, and the version number catches concurrent writers.

CREATED -> PROCESSING
PROCESSING -> REQUIRES_ACTION (OTP)
  -> AUTHORIZED | FAILED
AUTHORIZED -> CAPTURED | VOIDED
CAPTURED -> PARTIALLY_REFUNDED
  -> REFUNDED

UPDATE payments SET status = CAPTURED,
  version = version + 1
 WHERE id = :p AND status = AUTHORIZED

Deep dive: timeouts

The trickiest moment is a timeout. The provider may have charged the card, or not. Never mark it failed, and never retry with a new attempt id, which is how double charges happen. Keep the payment processing and resolve the unknown: ask the provider for the status of that attempt id, or wait for its webhook, retrying with exponential backoff. If it is still unknown after an hour, escalate to a human and keep the order on hold.

Deep dive: ledger

The ledger is where accounting lives. Every money movement is recorded as two or more entries that sum to zero: a capture debits what the customer owes and credits what we owe the seller plus our fee. A refund never edits the original; it adds new, reversing entries. Entries are append-only, so the ledger is a complete history, and any account's balance is just the sum of its entries, which you can cache but always rebuild. If the books do not sum to zero, you have found a bug.

capture 1,499.00 for seller 7:
  debit  customer_receivable 1,499.00
  credit seller_7_payable    1,424.05
  credit platform_fee           74.95
  sum = 0

refund: new entries, reversed
never UPDATE or DELETE an entry
balance = SUM(entries) per account
  cached as a snapshot, rebuildable

Deep dive: reconcile

Even with perfect code, messages get lost, so check reality every day. Providers send a settlement file listing every charge, refund, fee and payout. Match each line to our attempts by the provider's reference. Most match. A line only they have usually means a missed webhook, so apply it. A line only we have, or a different amount, goes to a review queue for the finance team. Reconciliation is the safety net that catches everything else.

Deep dive: card data

Handle card data by not handling it. Use the provider's SDK or hosted payment fields, so the card number goes straight from the browser to the provider and we receive only a token. Because our servers never see card numbers, our PCI compliance scope shrinks dramatically. The rest is still serious security: TLS everywhere, verified webhook signatures, secrets in a vault, and an audit log of every action support staff take.

Deep dive: providers

Relying on one provider makes their outage your outage. Integrate two, and route each payment by method, country, cost and live success rate. Put a circuit breaker on each provider: if errors spike, new payments go to the other one. But never move an attempt that is already in flight; resolve it where it started, or you risk charging twice. Background retries use exponential backoff with jitter and land in a dead letter queue after a few tries.

Trade-offs

Trade-offs. We often return processing and confirm later, because blocking a user on a thirty second bank call is worse. We use a strongly consistent relational store, accepting less write scale, because money cannot be eventually right. If our database is down, we stop taking payments rather than guess. Webhooks arrive at least once and sometimes out of order, and the state machine simply ignores moves that are not legal from the current state.

Monitor authorization success rate per provider and card type, because a sudden drop usually means a provider problem, not your users. Watch provider latency, the number of payments stuck in processing, daily reconciliation mismatches and the ledger imbalance, which must be exactly zero. At ten times the volume, shard payments by merchant and the ledger by account. New countries mostly mean new providers and payment methods, not a new architecture.

Interview summary

In the interview: low throughput, so correctness is the whole game. An idempotency key per business action, passed down to the provider. Write intent before calling out. An explicit state machine with conditional transitions. Treat a timeout as unknown and resolve it by querying or waiting. Record money in a double-entry, append-only ledger, reconcile against the provider daily, keep card data out with tokens, and route across two providers.