SYSTEM DESIGN · DESIGN PROBLEMS · PLATFORM SERVICES · HARD · 44 OF 51

Design a Recommendation System: Candidate Generation, Ranking, Cold Start and Feedback Loops

Design the "recommended for you" rows of a large app: the funnel from fifty million items to twenty, candidate sources with embeddings and nearest-neighbour search, a ranking model and its features, a feature store that keeps training and serving consistent, offline training versus online serving, cold start for new users and items, exploration, feedback loops and how to measure it.

Try it on the practice board Drag the blocks, wire them up, and check your design against the reference.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
  42. 42
  43. 43
  44. 44
  45. 45
  46. 46
  47. 47
  48. 48
  49. 49
  50. 50
  51. 51
1 / 18

Press play to hear the narration, or step through at your own pace.

Notes

Listen

Listen. We serve personal rows on the home page and similar items on each item page. Signals come from views, clicks, watch time, purchases and skips. New users and brand new items must still get sensible results. Some items must never appear: blocked, sold out, or already bought. Search ranking and the ads auction are separate systems, out of scope today.

The qualities. The whole row must come back within two hundred milliseconds at p ninety-nine. It should react within a session: watch two cooking videos and the next refresh leans to cooking. New items should be recommendable within an hour. If anything breaks, show popular items rather than an error. And success is measured by long-term value, like retention, not only clicks, which are easy to game with clickbait.

Functional
  • home: rows of personal picks
  • item page: "more like this"
  • learn from views, clicks,
  • watch time, buys, skips
  • new users and new items must
  • still get sensible results
  • never show blocked, sold out
  • or already-bought items
  • out of scope: search ranking,
  • ads auction
Non-functional
  • p99 < 200 ms for the whole row
  • react within the session:
  • watch a cooking video, see
  • more cooking minutes later
  • new items recommendable
  • within an hour
  • degrade, never fail: fall back
  • to popular items
  • measure by long-term value,
  • not only clicks

Ask scale

Scale. A hundred million daily users loading five pages with recommendations is five hundred million requests a day, about six thousand a second, seventeen thousand at peak. If each request scores five hundred candidates, that is eight and a half million model scores a second, so the ranker must be efficient and batched. Users generate twenty billion events a day, two terabytes. The catalog has fifty million items.

Now turn numbers into design. One hundred twenty-eight float numbers per item, for fifty million items, is about twenty-six gigabytes, or six and a half compressed to one byte each, which fits in memory on a few nearest-neighbour search nodes. User vectors are about a hundred fifty gigabytes, so they belong in the feature store. And no model can score fifty million items per request, so we need a funnel: about a thousand candidates, five hundred ranked, twenty shown.

Back of the envelope
100M daily users × 5 loads
  = 500M req/day ≈ 5,800/s
  peak ×3 ≈ 17,000/s
score 500 candidates each
  = 8.5M model scores/s at peak

events: 100M × 200 = 20B/day
  ≈ 230k/s × 100 B = 2 TB/day
catalog: 50M items
users: 300M with history
What the numbers say
item vectors: 50M × 128 × 4 B
  = 25.6 GB float32
  int8 → 6.4 GB, shard on a few
  ANN nodes, in memory
user vectors: 300M × 128 × 4 B
  ≈ 154 GB → feature store

cannot rank 50M per request
  → funnel: 50M → ~1,000
  → 500 ranked → 20 shown

Define API

The API. One read call takes the surface, the row, a limit and optional context, like the item being viewed, and returns items with a short reason, such as because you watched X. It also returns a request id. Clients send events in batches, impressions as well as clicks, each tagged with that request id and the position shown, so we can later tell what we showed from what the user chose.

API
GET /v1/recs?surface=home
    &row=for_you&limit=20
    &context=item_881
→ { request_id, items: [
    { id, reason, score } ] }

POST /v1/events  (batched)
  { request_id, item_id, type:
    impression|click|watch|buy,
    position, value, ts }

Data model

Data model. Raw events go to a data lake, partitioned by day, carrying the request id and model version. Features are keyed by user and by item: embeddings, recent items, top categories, click rates over different windows. They live twice, in an online key-value store for serving and offline in the lake for training. Models are versioned in a registry. And we log the exact features we scored with, which matters for training later.

Data model
events (lake, by day)
  user, item, type, pos, ts,
  request_id, model_version
features (online KV, offline lake)
  user:42 → embedding, top cats,
    last 50 items, counts 1h/7d
  item:881 → embedding, CTR 1d,
    age, price, category
models: version, metrics, owner
served log: features as scored

High-level design

Trace one request. The rec service asks several candidate sources for items, which return about a thousand, cheaply. A light pre-filter trims them to five hundred. The ranker fetches features for the user and those items in one batch, scores each one with a model, and returns the top hundred. A re-ranking step applies diversity and business rules and the final twenty go back. Budget roughly thirty, twenty, fifty and ten milliseconds for those steps.

The other half is the learning loop. Impressions and clicks land in the event log. A streaming job updates short-term features, like the last fifty items a user touched, within seconds, which is how the session stays fresh. Every day, the training pipeline builds labelled examples, trains a new model on GPUs, evaluates it offline and registers it. A good model rolls out as a canary, then as an experiment against the current one.

Deep dive: candidates

Deep dive one: candidates. Recall comes from several cheap sources. A two-tower model maps users and items into the same space, and an approximate nearest-neighbour index, such as HNSW, returns the items closest to the user's vector in milliseconds. Item-to-item lists come from co-occurrence: people who watched this also watched that. Add recently viewed, trending in the user's region, and followed creators. Merge, deduplicate, and cap each source so no one source floods the list.

Candidate sources
  • two-tower embeddings: user
  • vector → nearest items (ANN)
  • item-to-item: "people who
  • watched X also watched Y"
  • from co-occurrence counts
  • recent: continue, re-buy
  • trending by region and topic
  • followed creators or brands
  • merge, dedupe, cap per source
  • → ~1,000 items in < 30 ms

Deep dive: ranking

Deep dive two: ranking. The ranker sees far richer features than candidate generation: the user's history, item quality and freshness, cross features like whether this item matches the user's last five categories, and context like device and time of day. A gradient-boosted tree or a deep network predicts several outcomes: click, finish, buy, hide. A weighted blend turns them into one score, and those weights are a product decision, tuned with experiments.

The ranker
features per (user, item):
  user: history, embedding, locale
  item: CTR, age, price, quality
  cross: same category as last 5?
  context: device, hour, surface
model: boosted trees or a deep
  net, predicting p(click),
  p(finish), p(buy), p(hide)
score = 1.0·p(finish) + 0.3·p(buy)
        - 2.0·p(hide)

Deep dive: features

Deep dive three: features. The silent killer here is training-serving skew: a feature computed one way in the batch job and another way in the service, so the live model sees data it never trained on. A feature store defines each feature once and writes it to both the online and offline stores. Even better, log the exact features at serving time and train on those. When joining history, use point-in-time joins, so no example accidentally sees the future.

Avoiding training-serving skew
  • skew: a feature computed one
  • way in Spark, another in the
  • service → model sees new data
  • fix: define each feature once,
  • write to online and offline
  • log features at serve time;
  • train on what was really seen
  • point-in-time joins: no
  • features from the future
  • freshness: batch daily, stream
  • for session features

Deep dive: training

Deep dive four: offline versus online. Training is a batch job: retrain daily on the last thirty days, perhaps fine-tune hourly for a fast-moving catalog. Offline metrics like AUC or recall at K are a gate, not proof, because the logged data only shows items the old model chose to display. The real test is online: shadow traffic, a small canary, then an experiment against the current model, judged on watch time, purchases and retention.

Offline vs online
  • offline: daily retrain on 30
  • days of logs; hourly fine-tune
  • for fast-moving catalogs
  • offline metrics: AUC, recall@K
  • → only a gate, not proof
  • online: shadow → canary 5%
  • → A/B test vs current model
  • judge on watch time, buys,
  • 7-day retention
  • auto-rollback on bad latency

Deep dive: cold start

Deep dive five: cold start. A new user has no history, so start with popular items in their region, any interests chosen at sign-up, and context like device; after two or three clicks, session features take over. A new item has no clicks, so build its embedding from content, title, description and image, which places it near similar items in the index. Reserve a few exploration slots to show new items and learn quickly. For users with sparse history, lean on item-to-item lists.

Cold start and sparse data
  • new user: popular in region,
  • onboarding picks, device and
  • source of sign-up
  • → session signals after 2 clicks
  • new item: content embedding
  • from title, text, image
  • → near similar items in ANN
  • reserve exploration slots:
  • boost new items, measure fast
  • sparse users: lean on item-item

Deep dive: feedback loop

Deep dive six: the feedback loop. The model only learns from what it chose to show, so popular items get shown, clicked and shown more, and niche items never get a chance. Top positions also get clicks just for being on top, so log the position and model that bias. Spend about five percent of slots on exploration using a bandit method. Finally, re-rank for diversity, at most two items per creator, and filter items the user has already seen with a Bloom filter.

Feedback loops
  • popular gets shown → clicked
  • → more popular: rich get richer
  • position bias: top slots get
  • clicks just for being on top
  • → log position, model it
  • explore ~5% of slots
  • (Thompson sampling, bandits)
  • re-rank for diversity: max 2
  • per creator, mix categories
  • filter seen items (Bloom filter)

Trade-offs

Review the trade-offs. Precomputing recommendations nightly for every user is cheap but stale, and wasted on users who never come back. Running the whole funnel live is fresh but costs real compute. A hybrid caches candidates for an hour and ranks live. If the ranker is slow, skip it and serve candidates in source order; if everything fails, serve a cached popular list. Monitor latency, empty rows, click-through drift and feature freshness.

Trade-offs and failures
  • precompute nightly per user:
  • cheap, stale, wasted on idle
  • real-time funnel: fresh,
  • ~17k req/s of model compute
  • hybrid: cache candidates 1 h,
  • rank live
  • ranker slow → skip it, serve
  • candidate order; all down →
  • popular list from cache
  • monitor: p99, empty rows, CTR
  • drift, feature freshness

Interview summary

In the interview: a funnel from fifty million items to a thousand candidates, five hundred ranked and twenty shown. Candidates from embeddings with nearest-neighbour search, item-to-item, trending and recent. A ranker with rich features predicting several outcomes. A feature store so training matches serving. Train daily, prove it online. Handle cold start with content and popularity, and correct the feedback loop with exploration and diversity.

Say it like this
  • funnel: 50M → 1,000 → 500 → 20
  • candidates: two-tower + ANN,
  • item-item, trending, recent
  • ranker: rich features, multi-
  • outcome model, blended score
  • feature store: one definition,
  • log features at serve time
  • train daily, prove it online
  • cold start: content + popular
  • explore, debias, diversify

Interview questions

Why split recommendation into candidate generation and ranking?

No model can score tens of millions of items per request within a latency budget, so cheap sources first recall about a thousand plausible items and an expensive model then ranks only those, which balances recall, accuracy and cost.

How does embedding-based candidate generation work?

A two-tower model learns vectors for users and items in one space so that engaged pairs are close; at serving time an approximate nearest-neighbour index such as HNSW returns the items nearest to the user vector in a few milliseconds.

What is training-serving skew and how do you avoid it?

It is when features are computed differently for training than for serving, so the live model sees unfamiliar inputs. Define features once in a feature store that writes both online and offline copies, log features at serving time, and use point-in-time joins.

How do you handle a brand new user?

Start with popular items for their region and context plus any interests chosen at sign-up, then switch quickly to session-based signals after the first few interactions, before a long-term profile exists.

How do you recommend a brand new item with no interactions?

Build its embedding from content such as text, category and images so it lands near similar items, and reserve exploration slots that show new items to some users to gather feedback quickly.

What is a feedback loop in recommendations?

The model trains on interactions with items it chose to show, so popular items get more exposure and more clicks while others are never tested. Exploration, logging positions to correct position bias, and diversity rules counter it.

How do you evaluate a new ranking model?

Use offline metrics such as AUC or recall at K as a gate, then shadow traffic, a small canary and an online A/B test against the current model, judged on long-term metrics like watch time, purchases and retention.

What happens if the ranking service is slow or down?

Degrade instead of failing: return candidates in source order without ranking, or fall back to a cached popular or trending list, while timeouts keep the page within its latency budget.