Notes
Listen
Listen. Merchants register one or more HTTPS endpoints and choose event types, like payment succeeded or refund created. When an event happens, we POST it as JSON within seconds. If their server fails, we retry for days. Every request is signed so they can verify it came from us. A dashboard shows every attempt and lets them replay events, rotate secrets and disable endpoints.
The qualities. Delivery is at least once: we must never drop an accepted event, which means duplicates are possible, so receivers get a stable event id to deduplicate. Healthy endpoints should receive events within about five seconds. One slow or dead endpoint must not delay anyone else. We promise only best-effort ordering. And since merchants choose the URLs, we must never be tricked into calling our own internal network.
- merchants register endpoint
- URLs and pick event types
- POST each matching event as
- JSON within seconds
- retry until it succeeds or
- we give up (days later)
- signed so they can verify it
- dashboard: attempts, errors,
- manual and bulk replay
- rotate secrets, disable URLs
- at-least-once: never drop
- an accepted event
- duplicates possible → give
- receivers a stable event id
- p50 delivery < 5 s when healthy
- isolation: one slow endpoint
- must not delay others
- ordering: best effort only
- safe: no calls into our own
- network (SSRF)
Ask scale
Scale. Half a billion events a day, each going to one and a half endpoints on average, is seven hundred fifty million deliveries: about nine thousand a second, forty-three thousand at peak. Payloads average two kilobytes and are stored once per event, a terabyte a day, thirty terabytes for a thirty-day replay window. If two percent fail the first time, that is fifteen million retries a day, across three hundred thousand endpoints.
Here is the number that shapes the design. Requests in flight equal rate times latency. Healthy, forty-three thousand a second at three hundred milliseconds is about thirteen thousand open requests. Now one big merchant receiving five thousand a second starts timing out at ten seconds: that one endpoint alone wants fifty thousand requests in flight, four times the whole fleet's normal load. Without a cap per endpoint, one sick merchant starves everyone.
500M events/day × 1.5 endpoints = 750M deliveries/day ≈ 8,700/s avg, peak ×5 ≈ 43k/s payload 2 KB, stored once/event 500M × 2 KB = 1 TB/day keep 30 days ≈ 30 TB 2% fail first try ≈ 15M retries/day endpoints: 300k across 200k merchants
in flight = rate × latency healthy: 43k/s × 0.3 s ≈ 13,000 open requests one big merchant, 5k/s, times out at 10 s: 5k × 10 s = 50,000 in flight → 4× the whole fleet's normal → cap concurrency per endpoint
Define API
Two sides to the API. Merchants use ours to register an endpoint and receive its signing secret, list attempts, and replay events. Then there is the request we send them: a POST with the event as JSON and three headers: the event id, a timestamp, and a signature. That shape is a public contract, so version it and never change it silently. Success means any two hundred status within ten seconds.
POST /v1/endpoints
{ url, events: [...] } → secret
POST /v1/events/{id}/replay
GET /v1/endpoints/{id}/attempts
we send:
POST https://shop.example/hooks
Webhook-Id: evt_7Hq2
Webhook-Timestamp: 1763200000
Webhook-Signature: v1=9f2c... Data model
Data model. Endpoints are small, relational config: URL, event types, one or two active secrets and a status. Each event's payload is stored once. A delivery row links one event to one endpoint, with its state, attempt count and next attempt time, which is indexed so the scheduler can find due retries. Each attempt adds a small row with the status code, latency and the first bytes of the response, which powers the dashboard.
endpoints(id, merchant, url, events[], secrets[], status) events(id, type, merchant, payload, created_at) 30 days deliveries(id, event_id, endpoint_id, state, attempts, next_attempt_at) attempts(delivery_id, n, code, latency_ms, body_snippet, at) index: (state, next_attempt_at)
High-level design
Trace one event. The payment service commits the payment and an outbox row in one transaction, and a relay publishes it to Kafka. The dispatcher looks up which endpoints subscribe to this event type for this merchant, from a cached copy of the subscriptions table, and writes one delivery per endpoint to a queue partitioned by endpoint id. A worker signs the payload, POSTs it, and records the attempt.
When a POST fails, the worker does not sleep and retry inline, which would block the partition. It records the failure, computes the next attempt time and hands the delivery to the retry scheduler. The scheduler keeps an index of deliveries by due time and puts each one back on the queue when it is due. After the last retry, the delivery moves to dead letters, where merchants can replay it.
Deep dive: at-least-once
Deep dive one: at-least-once. Every hand-off only acknowledges after the next step is durable. The producer writes an outbox row in the same transaction as the business change, so an event cannot be lost between them. The dispatcher commits its Kafka offset only after the deliveries are saved. The worker marks a delivery done only after a two hundred response. A crash anywhere re-runs a step, which creates a duplicate but never a loss.
- producer: outbox row in the
- same transaction as the
- payment, relay to Kafka
- dispatcher: commit Kafka offset
- only after deliveries saved
- worker: mark delivered only
- after a 2xx response
- crash anywhere → step re-runs
- → a duplicate, never a loss
- so: same event id every time
Deep dive: retries
Deep dive two: retries. Use exponential backoff: one minute, two, four, eight, doubling up to a cap of twelve hours, for about fifteen attempts over three days. Multiply each delay by a random factor, called jitter, so that when a big merchant recovers, thousands of queued retries do not hit them in the same second. Retry timeouts, five hundreds and rate limits, honour a Retry-After header, and treat four ten gone as a request to disable the endpoint.
delay = min(base × 2^n, 12 h)
× random(0.5, 1.5) (jitter)
n: 1 2 3 4 5 ...
1m 2m 4m 8m 16m ...
~15 attempts over ~3 days
retry: timeout, 5xx, 429, 408
429 + Retry-After → honour it
410 Gone → disable endpoint
then: dead letter + email Deep dive: signing
Deep dive three: signing. Each endpoint has a secret. We compute an HMAC with SHA-256 over the event id, the timestamp and the raw body, and send it in a header. The receiver recomputes it with the same secret and compares in constant time. Because the timestamp is signed, the receiver can reject anything older than five minutes, which stops an attacker replaying a captured request. For rotation, keep two secrets active for a day and send both signatures.
msg = id + "." + timestamp + "." + body
sig = HMAC-SHA256(secret, msg)
receiver:
recompute, constant-time compare
reject if |now - ts| > 5 min
(stops replayed requests)
rotation: 2 secrets for 24 h,
send both signatures
never put secrets in the URL Deep dive: idempotency
Deep dive four: duplicates and order. Since we deliver at least once, every retry and replay of an event carries the same id, and we tell receivers to insert that id under a unique constraint and skip repeats. Ordering is not guaranteed: a refund webhook can arrive before the payment one, after retries. So include the resource's version and creation time, or suggest receivers treat the webhook as a hint and fetch the latest state from our API.
- same Webhook-Id on every
- retry and replay
- receiver: INSERT event id
- UNIQUE, skip if exists
- order not guaranteed:
- refund can beat payment
- → include resource version
- and created_at
- → or receiver fetches latest
- state from our API
Deep dive: isolation
Deep dive five: noisy neighbours. We cannot create a queue per endpoint for three hundred thousand endpoints, so partition by endpoint id and enforce fairness in the worker. Each endpoint gets a cap, say twenty requests in flight. When an endpoint is at its cap or failing, its deliveries are parked in the retry store so the partition keeps moving for everyone else. A circuit breaker pauses an endpoint that mostly fails, and probes it once a minute until it recovers.
- cannot have a Kafka topic per
- endpoint (300k of them)
- partition by endpoint id;
- worker caps in-flight per
- endpoint (e.g. 20)
- at the cap → park delivery in
- the retry store, move on
- circuit breaker: 50% failing
- → pause, probe every minute
- per-endpoint rate limit option
Deep dive: security
Deep dive six: security. The merchant chooses the URL, so assume someone will enter an internal address hoping our workers call it from inside our network. Allow HTTPS only, resolve the hostname and block private, loopback and cloud metadata ranges, and check again at connect time to defeat DNS rebinding. Send all traffic through egress proxies with fixed public IPs, which also lets merchants allowlist us. Do not follow redirects.
- HTTPS only, valid certificates
- resolve DNS, then block private
- and link-local ranges
- (10.x, 169.254.x, 127.x ...)
- check again at connect time
- (DNS rebinding)
- send through egress proxies
- with fixed public IPs
- → merchants can allowlist
- no redirects; 10 s timeout
Deep dive: observability
Deep dive seven: observability, for two audiences. Merchants see every attempt with the status code, latency and a snippet of their response, and can replay one event or everything since a time after fixing a bug. We watch success rate per endpoint, the age of the oldest undelivered event, the retry backlog and latency. Queue age is the key alert. After a day of failures, email the merchant; after three, disable the endpoint.
- merchant view: every attempt,
- code, latency, response body
- snippet; replay one or all
- since a time
- our metrics: success rate per
- endpoint, oldest undelivered
- age, retry backlog, p99
- alerts: queue age > 1 min,
- global success rate drop
- email merchant after 1 day down
Trade-offs
Review the trade-offs. Exactly-once delivery over the internet is not achievable, so we promise at least once plus stable ids, and let receivers make processing idempotent. Dropping strict ordering buys throughput and isolation. Sending full payloads saves merchants an API call, while thin events with only ids are smaller and never stale. If the retry store fails, workers hold failed deliveries briefly while healthy deliveries keep flowing.
- at-least-once + ids instead of
- exactly-once: simpler, honest
- no strict order: far more
- throughput and isolation
- thin events (ids only) vs full
- payloads: less data, more
- API calls back to us
- retry store down → workers
- pause failures, healthy
- deliveries keep flowing
Interview summary
In the interview: about forty-three thousand deliveries a second at peak. An outbox into Kafka, fanned out to one delivery per endpoint, each step acknowledging only after the next is durable. Exponential backoff with jitter for three days, then dead letters with replay. HMAC signatures with a timestamp, and the same event id on every retry. Cap concurrency per endpoint so one slow merchant cannot hurt others, and block private addresses.
- 750M/day, 43k/s peak
- outbox → Kafka → per-endpoint
- deliveries, ack after durable
- backoff 1m doubling, jitter,
- ~3 days, then dead letter
- HMAC over id.ts.body, 5 min
- same event id on every retry
- cap in-flight per endpoint,
- park slow ones, breakers
- block private IPs, fixed egress