Design a Notification System: Push, SMS and Email at a Billion a Day

Listen

Listen. Internal services send to a user on push, SMS, email or an in-app inbox, now or at a scheduled time, to one user or a segment of millions. Messages come from templates with variables and languages. We must respect opt-outs and quiet hours, and track sent, delivered and opened. Building the providers themselves is out of scope.

The qualities. Login codes and security alerts must arrive within about five seconds, even during a huge marketing send. Marketing can wait minutes. Internally we can deliver at least once, but the user should see each message only once. Opt-outs are a legal requirement, so never break them. And a provider outage must not stop critical messages.

Ask scale

Scale. A hundred million daily users getting ten notifications each is a billion a day, about ten thousand a second. One campaign to fifty million users in an hour adds fourteen thousand a second, so plan for a peak around fifty thousand. The delivery log is about half a terabyte a day; keeping thirty days is fifteen terabytes.

100M daily users
10 notifications/user/day
  = 1B/day ≈ 10,000/s average

campaign: 50M users in 1 hour
  ≈ 14,000/s extra
peak ≈ 50,000/s

delivery log: 1B × 500 B
  = 500 GB/day, keep 30 days
  ≈ 15 TB

Define API

The API. One send call takes an idempotency key, a user or a segment, a template and its parameters, a priority and an optional send time. It returns two oh two accepted, because delivery is asynchronous. Other endpoints read status, update preferences, register device tokens, and receive delivery receipts from providers as webhooks.

POST /v1/notifications
  { idempotency_key, user_id |
    segment_id, template_id,
    params, priority, send_at }
  → 202 { notification_id }

GET  /v1/notifications/{id}
PUT  /v1/users/{id}/preferences
POST /v1/users/{id}/devices
POST /v1/webhooks/{provider}

Data model

Data model. Devices, preferences and templates are modest, relational and read constantly, so keep them in SQL with a cache in front. Preferences are per category and channel, plus quiet hours and a time zone. The delivery log is huge and append-heavy, so it goes in a wide-column store like Cassandra, keyed by notification id.

devices (SQL)
  user_id, token, platform, seen
preferences (SQL)
  user_id, category, channel, on
  quiet_hours, timezone
templates (SQL)
  id, version, channel, locale

deliveries (Cassandra)
  PK notification_id, channel
  status, provider_msg_id, ts

High-level design

The high-level design. The API validates the request, checks the idempotency key, looks up preferences and device tokens, renders the template, and puts one job per channel on a queue. It replies right away. Channel workers pull jobs and call the right provider: Apple and Google push services, an SMS gateway or an email service.

Two more pieces. Workers record every attempt in the delivery log, and providers call back later with delivered, bounced or opened, which updates the same record. A scheduler holds messages with a future send time and campaign jobs, and releases them onto the queues when they are due, in the user's own time zone.

Deep dive: priority

Deep dive one: priority. With a single queue, a fifty million message sale campaign goes in at eight pm, and a login code sent one second later waits behind all of it, forty minutes. The user cannot log in. This is the most common failure in real notification systems, and the fix is isolation.

Give each priority and channel its own queue and its own workers. Critical, like login codes and security alerts, has dedicated workers that bulk traffic can never use. Normal covers order updates. Bulk is marketing, and is rate-capped. Providers have their own limits, say five hundred SMS a second, so a token bucket per provider reserves a share for critical messages.

Deep dive: reliability

Deep dive two: reliability. Timeouts and five hundred errors are retried with exponential backoff and jitter, so retries do not stampede. If a provider keeps failing, a circuit breaker switches critical SMS to a backup provider, see Concepts episode 40. Permanent errors, like an invalid number, are not retried. After the maximum attempts, the job goes to a dead-letter queue for inspection.

Queues deliver at least once, so the same job can arrive twice. Deduplicate at two points. At the API, a repeated idempotency key returns the existing notification. In the worker, skip a message whose sent marker exists, and set the marker once the provider accepts it; a crash in between can rarely repeat one. Concepts episode 29 covers this. Push also supports a collapse key, so a duplicate replaces rather than stacks.

A quieter problem: dead device tokens. People uninstall apps and switch phones. Apple and Google answer with unregistered or a gone status for those tokens. Delete them right away, and expire tokens not seen in a couple of months. Otherwise a large share of your push traffic is wasted, and providers may throttle you.

API: idempotency_key → id
  (unique index, 24 h)
  same key again → same id

worker, before sending:
  EXISTS sent:{id}:{ch} → skip
after the provider accepts:
  SET sent:{id}:{ch} EX 86400
crash in between: rare repeat
push: collapse key, so a
  retry replaces, not repeats

Deep dive: campaigns

Deep dive three: campaigns. A segment of fifty million users is resolved to a list of ids in object storage and split into batches of a thousand. Each batch job drops opted-out users, holds back anyone in quiet hours until morning in their time zone, applies a frequency cap such as two marketing pushes a day, renders the template and enqueues on the bulk queues, paced to finish in an hour.

Trade-offs

Failure modes. A provider outage trips the breaker and critical traffic moves to a backup. A queue backlog pauses bulk sends while critical continues. If the preferences store is down, use cached preferences, and when unsure, skip marketing but still send transactional messages. A crashed worker's job reappears after its visibility timeout, and the sent marker prevents a repeat.

Monitor the age of the oldest message in each queue, the p ninety-nine time to deliver a login code, provider errors and latency, delivery and open rates, and spikes in opt-outs, which mean you are annoying people. At ten times the volume, add queue partitions and workers, contract several providers per channel, and deploy per region.

Say it like this

In the interview: one API that accepts quickly with an idempotency key. Preferences and templates in SQL, the delivery log in a wide-column store. Queues per priority and channel, with dedicated workers for critical messages, and a token bucket per provider. Retries with backoff, circuit breakers and backup providers, a sent marker to avoid repeats, and campaigns that are batched, capped and paced.