Design Twitter: Tweets, Home Timelines, Real-Time Search and Trends

Listen

Requirements. Post a tweet of up to two hundred eighty characters, maybe with a media reference. Follow people. See two timelines: the home timeline, built from everyone you follow, and a user timeline, one person's own tweets. Search recent tweets and see trending topics. Direct messages, ads, audio rooms and editing are out of scope for today.

Qualities. The home timeline should return in under two hundred milliseconds. A tweet should reach followers within a few seconds and be searchable within about ten. Reads dominate. Eventual consistency is fine: nobody minds seeing a tweet two seconds late. And traffic is spiky: a final match or an election can multiply tweets per second several times in a minute, so plan headroom.

Ask scale

Assume two hundred fifty million daily users and five hundred million tweets a day. That's about five thousand tweets a second on average, and five times that during a big event. Each user loads the timeline twenty times a day: five billion loads, fifty thousand a second, a hundred fifty thousand at peak. With around two hundred followers each, every tweet fans out to roughly two hundred timelines.

Storage. A tweet with its metadata is about a kilobyte, so five hundred gigabytes a day, about a hundred eighty terabytes a year before replication. Media lives in object storage behind a CDN, a separate problem. And the timeline cache is small: eight hundred tweet IDs of eight bytes for two hundred fifty million active users is about one point six terabytes. A Redis cluster holds that comfortably.

DAU                 250M
tweets/day          500M
  500M / 100,000 s = 5K wps
  big-event peak   = 25K+ wps

timeline loads      20/user/day
  5B / 100,000 s   = 50K rps
  peak             = 150K rps

read:write          ~10 to 100:1
avg followers       ~200

tweet: text 280 chars UTF-8
  + ids, ts, counts  ~ 1 KB
500M x 1 KB      = 500 GB/day
  x 365          = ~180 TB/yr
  x 3 replicas   = ~550 TB/yr

timeline cache: 800 IDs x 8 B
  x 250M active  = 1.6 TB

media: separate blob store

Define API

The API. Post a tweet with text, optional media IDs and an optional reply-to. Two timeline reads, home and user, both paginated with max ID: give me twenty tweets older than this ID. That works because tweet IDs sort by time. Search takes a query and a since ID for polling new results. Trends take a place. And follow is a simple POST. Writes carry an idempotency key so a retry never posts twice.

POST /v1/tweets
  { text, mediaIds[], replyTo? }
  -> { tweetId }
GET  /v1/timeline/home
     ?max_id=..&count=20
GET  /v1/users/{id}/tweets
     ?max_id=..&count=20
GET  /v1/search?q=..&since_id=..
GET  /v1/trends?place=..
POST /v1/users/{id}/follow

Data model

Data model, starting with IDs. Use a sixty-four-bit ID with a millisecond timestamp in the high bits, then a worker number and a sequence, the scheme Twitter published as Snowflake. IDs are unique without coordination and sort by time, so newest-first is just descending ID order. Tweets are sharded by tweet ID, the user timeline is an index by author, follows are stored in both directions, and the home timeline is a capped Redis list per user.

tweetId 64-bit:
  41 bits ms time | 10 bits
  worker | 12 bits sequence
tweet  (tweetId PK, authorId,
        text, replyTo, ts)
user_tl authorId -> [tweetId]
follow (follower, followee)
       stored both directions
home_tl userId -> Redis list
counts tweetId -> likes, RTs

High-level design

The write path. The tweet service gets an ID, persists the tweet, and publishes an event. Fan-out workers read the author's followers from the graph and push the tweet ID onto each follower's home timeline list in Redis, trimming the list to eight hundred so memory stays bounded. The client got its success response right after the persist; fan-out happens in the background.

The read path. The timeline service reads the first twenty IDs from your Redis list, multi-gets the tweets from a tweet cache in front of the store, adds author info and counts, and returns. That's one list read and one batched lookup. Notice the same event stream also feeds the search indexer and the trends counter, so each consumer scales independently and a slow indexer never delays timelines.

Deep dive: timelines

Deep dive one: timelines and the celebrity problem. A tweet from an account with fifty million followers would mean fifty million list pushes, which takes minutes and swamps the workers just when everyone wants that tweet. So above a follower threshold we don't push. At read time, the timeline service takes your precomputed list and merges in the recent tweets of the few celebrities you follow, which are hot in cache. Sort by ID, which is time order, and return.

A few edge cases interviewers like. Don't fan out to users who haven't logged in for a month; rebuild their timeline when they return. On unfollow, filter that author out at read time and clean the list lazily, instead of scanning it immediately. Deletes leave a tombstone in the tweet store, so hydration simply skips the deleted ID. The cached list is a hint; the store is the truth.

Deep dive: search

Deep dive two: search. An indexer consumes the stream, breaks each tweet into terms, hashtags and mentions, and appends the tweet ID to each term's posting list in an inverted index. Partition the index by time: one hot segment takes all new writes and is searched first, older segments are read-only. A query fans out to the shards, takes the best results from each, and merges them by recency and engagement. Twitter has described a system like this, called Earlybird.

Deep dive: trends

Deep dive three: trends. A stream processor counts hashtags and phrases in one-minute buckets per region. Exact counts for every phrase are too expensive, so use a count-min sketch for approximate counts and a heap for the top candidates. Crucially, trending isn't the same as popular: score each term by how much its current rate exceeds its usual rate, otherwise common words always win. Filter spam and refresh every minute or so.

Trade-offs

Trade-offs and failures. If fan-out falls behind, timelines lag, but the durable stream means no tweet is lost. If a Redis node dies, rebuild those lists by pulling from followees. A viral tweet is a hot key in the tweet cache, so replicate it to several nodes. During a big event, autoscale workers and you can lower the celebrity threshold on the fly. Watch fan-out lag, p ninety-nine timeline latency, search index lag and cache hit rate.

Say the trade-offs out loud. Pushing costs memory and write throughput but makes reads one cache hit. Pulling for celebrities costs a little read latency but keeps writes sane. Eventual consistency buys availability. At ten times the scale, add regions, shard Redis by user ID, keep fewer IDs per list, and move old tweets to cheaper storage since almost nobody reads them.

Say it like this

The summary. Tweets get time-sortable sixty-four-bit IDs and are stored sharded by ID. After persisting, an event drives three consumers: fan-out into capped Redis home timelines, a time-partitioned inverted index for search, and a windowed counter for trends. Celebrities are pulled and merged at read time. Everything downstream is eventually consistent, which is exactly what this product needs.