Design a Digital Wallet: Atomic Transfers, Cross-Shard Money and Hot Accounts

Listen

Scope it. Users top up from a card or bank, send money to another wallet, pay merchants, sometimes with a temporary hold, withdraw to a bank, and see their balance and history. The non-functional rules are strict: a balance can never go negative, money is never created or lost, a transfer completes in a few hundred milliseconds and the service is always on. Currency exchange and credit lines are out of scope; we assume a single currency.

Ask scale

A hundred million users, twenty million active each day, doing five transactions each is a hundred million a day: about twelve hundred transfers a second on average and twelve thousand at a festival peak. Design with headroom for fifty thousand. Balance checks are ten times more frequent. Each transfer writes a transaction row and two ledger entries, around a kilobyte, so a hundred gigabytes a day and tens of terabytes a year, kept for a decade. One database cannot take that write load, so we will shard.

100M users, 20M daily active
20M x 5 transactions = 100M/day
  = 1,200 TPS avg
  festival peak x10 = 12,000 TPS
  design for 50,000 TPS headroom
balance reads about 10x writes

per transfer: 1 txn + 2 entries
  about 1 KB -> 100 GB/day
  = 36 TB/year, kept 10 years

Define API

The API is small. Get my wallet, returning balance, held and available. Create a transfer to another wallet with an amount in paise and an idempotency key, so a retry after a dropped connection cannot send the money twice. Top-ups and withdrawals go through the payment system, from Design Problems thirty. And a paginated history. Every endpoint that moves money is idempotent.

GET  /v1/wallets/me
  -> { balance, held, available }
POST /v1/transfers
  Idempotency-Key: 3c9e...
  { to: "w_88", amount: 50000,
    note: "dinner" }
  -> 201 { txn_id, status: DONE }
POST /v1/topups      { amount, method }
POST /v1/withdrawals { amount, bank }
GET  /v1/wallets/me/history?cursor=

Data model

Each wallet row holds its current balance, the amount on hold, a version and its KYC tier. Every change also appends a ledger entry with a signed amount and the balance after it, so the balance is always explainable. A transfers table records each transaction with its idempotency key. Use a relational database for transactions and constraints, and shard by wallet id across, say, sixty-four shards. The global invariant: all entries across all wallets sum to zero.

wallets(id, user_id, balance, held,
  version, status, kyc_tier)
entries(id, txn_id, wallet_id,
  amount (+ or -), balance_after, at)
  append only
transfers(txn_id, idem_key UNIQUE,
  from, to, amount, status, at)

shard = hash(wallet_id) % 64
invariant: sum of all entries = 0

High-level design

Trace a transfer. The app calls the gateway, which authenticates the user and the device. The wallet service checks the idempotency key, then asks the limits and risk service whether this transfer fits the sender's KYC tier and velocity limits. Then it moves the money: debit the sender, credit the receiver, write both entries. When both wallets are on the same shard, that is one ordinary database transaction.

Deep dive: same shard

Within one shard the transfer is a single transaction. The debit is conditional: subtract the amount only where the balance minus holds is at least that much. If zero rows change, roll back and report insufficient funds; a negative balance is simply impossible. Then credit the receiver and write both entries. One more detail interviewers love: always lock the two wallets in a fixed order, say lower id first, or two people paying each other at once can deadlock.

BEGIN;
UPDATE wallets SET balance = balance
  - 50000 WHERE id = A
  AND balance - held >= 50000;
-- 0 rows -> ROLLBACK, insufficient
UPDATE wallets SET balance = balance
  + 50000 WHERE id = B;
INSERT entries (A, -50000), (B, +50000);
COMMIT;
lock order: lower wallet id first

Deep dive: cross shard

Most transfers cross shards. Use a saga with a clearing account. Step one, on the sender's shard, in one local transaction: debit the sender, credit clearing, and write a credit-receiver message to an outbox. Step two, on the receiver's shard: debit clearing and credit the receiver, deduplicated on the transaction id. If the receiver's wallet is closed, compensate by refunding the sender. The books balance at every instant.

Be ready to compare. Two-phase commit makes the transfer atomic across shards, but it holds locks across a network round trip, and if the coordinator dies at the wrong moment both shards wait. The saga uses only local transactions, so it is fast and never blocks, at the cost of the receiver seeing the money a few milliseconds later and the need for a compensation path. For wallets, the saga with a clearing account is the usual choice. See Concepts episode 33.

step 1 (shard of A), one txn:
  debit A, credit clearing,
  outbox: CreditB(txn_id)
step 2 (shard of B), one txn:
  debit clearing, credit B
  dedupe on txn_id
B closed? compensate:
  debit clearing, refund A

clearing balance = money in flight

Deep dive: single writer

For far more throughput, change the model. Route every command for a wallet to one Kafka partition, handled by a single processor holding balances in memory. With only one writer per wallet, it validates and applies commands with no locks, emitting debited and credited events. Events become the source of truth and the database is a projection. After a crash, load a snapshot and replay. Cross-wallet transfers still use the saga, as messages between processors.

commands partitioned by wallet id
  -> Kafka partition -> one processor
processor keeps balances in memory
  validates: available >= amount
  emits events: Debited, Credited
events are the source of truth
  DB rows are a projection
crash: reload snapshot + replay

one thread per partition, no locks

Deep dive: hot accounts

Now the hot account. A large merchant may receive five thousand payments a second, all crediting one row, and that row's lock becomes a queue. Split the merchant into sixteen sub-wallets and credit a random one; the balance is their sum. Credits never fail for lack of funds, so this is safe. Payouts are rare, so sweep the sub-wallets together first. Another option is batching: add up a hundred milliseconds of credits and apply them in one update.

5,000 payments/s into one row
  -> one row lock -> queue

split into 16 sub-wallets
  credit a random sub-wallet
  balance = sum of the 16
debits (payouts) are rare:
  sweep all subs into one, then pay
or batch: sum credits every 100 ms
  into one update

Deep dive: holds

Some payments need a hold, like a fuel pump or a cab ride whose final price is not known yet. Authorizing increases held, but only if the available amount covers it, and records the hold with an expiry. Capturing reduces the balance by the final amount and releases the hold. A void or an expiry just releases it. Available is always balance minus held, and every step is a conditional, idempotent update.

authorize: held = held + 30000
  WHERE balance - held >= 30000
  hold(id, wallet, amount, expires)
capture: balance - 25000,
  held - 30000 (final price 250)
void or expiry: held - 30000

available = balance - held
all transitions conditional, idempotent

Deep dive: limits

Wallets are regulated, so limits run before any money moves. Each KYC tier caps the balance, monthly inflow and per-transfer amount. Velocity rules watch for many transfers in an hour, a new device or a brand new payee. These synchronous checks must take under about thirty milliseconds, so keep the counters in Redis. Heavier fraud models run asynchronously on the event stream and can freeze a wallet afterwards. Freezing is a status change, never a deletion.

Deep dive: reads

Reads come in two kinds. The balance must reflect the transfer you just made, so read it from the shard's primary, or a replica that is known to be caught up. History is different: stream ledger entries through change data capture into a wide-column store keyed by wallet id and time, which pages efficiently and keeps that load off the money database. A second of lag in history is fine; a second of lag in the balance generates support tickets.

Deep dive: reconcile

The money behind the wallets sits in a real bank account held on behalf of users. So every night, check the big invariant: the sum of all wallet balances, plus money in clearing and pending payouts, must equal the bank account's balance. Also check that every transaction's entries sum to zero and every wallet's balance equals its last entry's balance after. A mismatch stops the affected process and pages a human. This is how you prove nothing was created or lost.

real money sits in a bank pool
  account held for users

nightly checks:
  sum(all wallet balances)
    + clearing + pending payouts
    = bank pool balance
  every txn: entries sum to 0
  balance = last balance_after
mismatch -> freeze job, page

Trade-offs

Trade-offs. Sharded SQL is simple and proven, and handles tens of thousands of transfers a second, which covers this problem. Single-writer event sourcing reaches much higher throughput, but is harder to build, debug and operate, so propose it only when the numbers demand it. The saga leaves money briefly in flight. If a shard fails, only its wallets pause. Step two of a saga retries until it succeeds, and an alert fires if anything sits in clearing too long.

Monitor transfer latency at p ninety-nine, failure reasons such as insufficient funds or limits, the clearing balance and how old its oldest item is, lock waits on hot rows, and the nightly reconciliation, which must match exactly. At ten times the load, add shards and more sub-wallets for hot merchants. At a hundred times, move to single-writer processors.

Interview summary

In the interview: twelve thousand transfers a second at peak, so shard SQL by wallet id. Store balances plus an append-only ledger. Debit conditionally so a balance can never go below zero, locking in a fixed order. Cross-shard transfers run as a saga through a clearing account with an outbox. Split hot merchants into sub-wallets. Support holds and limits, serve history from a read model, and reconcile every night against the bank.