Notes
Listen
Listen. We serve personal rows on the home page and similar items on each item page. Signals come from views, clicks, watch time, purchases and skips. New users and brand new items must still get sensible results. Some items must never appear: blocked, sold out, or already bought. Search ranking and the ads auction are separate systems, out of scope today.
The qualities. The whole row must come back within two hundred milliseconds at p ninety-nine. It should react within a session: watch two cooking videos and the next refresh leans to cooking. New items should be recommendable within an hour. If anything breaks, show popular items rather than an error. And success is measured by long-term value, like retention, not only clicks, which are easy to game with clickbait.
- home: rows of personal picks
- item page: "more like this"
- learn from views, clicks,
- watch time, buys, skips
- new users and new items must
- still get sensible results
- never show blocked, sold out
- or already-bought items
- out of scope: search ranking,
- ads auction
- p99 < 200 ms for the whole row
- react within the session:
- watch a cooking video, see
- more cooking minutes later
- new items recommendable
- within an hour
- degrade, never fail: fall back
- to popular items
- measure by long-term value,
- not only clicks
Ask scale
Scale. A hundred million daily users loading five pages with recommendations is five hundred million requests a day, about six thousand a second, seventeen thousand at peak. If each request scores five hundred candidates, that is eight and a half million model scores a second, so the ranker must be efficient and batched. Users generate twenty billion events a day, two terabytes. The catalog has fifty million items.
Now turn numbers into design. One hundred twenty-eight float numbers per item, for fifty million items, is about twenty-six gigabytes, or six and a half compressed to one byte each, which fits in memory on a few nearest-neighbour search nodes. User vectors are about a hundred fifty gigabytes, so they belong in the feature store. And no model can score fifty million items per request, so we need a funnel: about a thousand candidates, five hundred ranked, twenty shown.
100M daily users × 5 loads = 500M req/day ≈ 5,800/s peak ×3 ≈ 17,000/s score 500 candidates each = 8.5M model scores/s at peak events: 100M × 200 = 20B/day ≈ 230k/s × 100 B = 2 TB/day catalog: 50M items users: 300M with history
item vectors: 50M × 128 × 4 B = 25.6 GB float32 int8 → 6.4 GB, shard on a few ANN nodes, in memory user vectors: 300M × 128 × 4 B ≈ 154 GB → feature store cannot rank 50M per request → funnel: 50M → ~1,000 → 500 ranked → 20 shown
Define API
The API. One read call takes the surface, the row, a limit and optional context, like the item being viewed, and returns items with a short reason, such as because you watched X. It also returns a request id. Clients send events in batches, impressions as well as clicks, each tagged with that request id and the position shown, so we can later tell what we showed from what the user chose.
GET /v1/recs?surface=home
&row=for_you&limit=20
&context=item_881
→ { request_id, items: [
{ id, reason, score } ] }
POST /v1/events (batched)
{ request_id, item_id, type:
impression|click|watch|buy,
position, value, ts } Data model
Data model. Raw events go to a data lake, partitioned by day, carrying the request id and model version. Features are keyed by user and by item: embeddings, recent items, top categories, click rates over different windows. They live twice, in an online key-value store for serving and offline in the lake for training. Models are versioned in a registry. And we log the exact features we scored with, which matters for training later.
events (lake, by day)
user, item, type, pos, ts,
request_id, model_version
features (online KV, offline lake)
user:42 → embedding, top cats,
last 50 items, counts 1h/7d
item:881 → embedding, CTR 1d,
age, price, category
models: version, metrics, owner
served log: features as scored High-level design
Trace one request. The rec service asks several candidate sources for items, which return about a thousand, cheaply. A light pre-filter trims them to five hundred. The ranker fetches features for the user and those items in one batch, scores each one with a model, and returns the top hundred. A re-ranking step applies diversity and business rules and the final twenty go back. Budget roughly thirty, twenty, fifty and ten milliseconds for those steps.
The other half is the learning loop. Impressions and clicks land in the event log. A streaming job updates short-term features, like the last fifty items a user touched, within seconds, which is how the session stays fresh. Every day, the training pipeline builds labelled examples, trains a new model on GPUs, evaluates it offline and registers it. A good model rolls out as a canary, then as an experiment against the current one.
Deep dive: candidates
Deep dive one: candidates. Recall comes from several cheap sources. A two-tower model maps users and items into the same space, and an approximate nearest-neighbour index, such as HNSW, returns the items closest to the user's vector in milliseconds. Item-to-item lists come from co-occurrence: people who watched this also watched that. Add recently viewed, trending in the user's region, and followed creators. Merge, deduplicate, and cap each source so no one source floods the list.
- two-tower embeddings: user
- vector → nearest items (ANN)
- item-to-item: "people who
- watched X also watched Y"
- from co-occurrence counts
- recent: continue, re-buy
- trending by region and topic
- followed creators or brands
- merge, dedupe, cap per source
- → ~1,000 items in < 30 ms
Deep dive: ranking
Deep dive two: ranking. The ranker sees far richer features than candidate generation: the user's history, item quality and freshness, cross features like whether this item matches the user's last five categories, and context like device and time of day. A gradient-boosted tree or a deep network predicts several outcomes: click, finish, buy, hide. A weighted blend turns them into one score, and those weights are a product decision, tuned with experiments.
features per (user, item):
user: history, embedding, locale
item: CTR, age, price, quality
cross: same category as last 5?
context: device, hour, surface
model: boosted trees or a deep
net, predicting p(click),
p(finish), p(buy), p(hide)
score = 1.0·p(finish) + 0.3·p(buy)
- 2.0·p(hide) Deep dive: features
Deep dive three: features. The silent killer here is training-serving skew: a feature computed one way in the batch job and another way in the service, so the live model sees data it never trained on. A feature store defines each feature once and writes it to both the online and offline stores. Even better, log the exact features at serving time and train on those. When joining history, use point-in-time joins, so no example accidentally sees the future.
- skew: a feature computed one
- way in Spark, another in the
- service → model sees new data
- fix: define each feature once,
- write to online and offline
- log features at serve time;
- train on what was really seen
- point-in-time joins: no
- features from the future
- freshness: batch daily, stream
- for session features
Deep dive: training
Deep dive four: offline versus online. Training is a batch job: retrain daily on the last thirty days, perhaps fine-tune hourly for a fast-moving catalog. Offline metrics like AUC or recall at K are a gate, not proof, because the logged data only shows items the old model chose to display. The real test is online: shadow traffic, a small canary, then an experiment against the current model, judged on watch time, purchases and retention.
- offline: daily retrain on 30
- days of logs; hourly fine-tune
- for fast-moving catalogs
- offline metrics: AUC, recall@K
- → only a gate, not proof
- online: shadow → canary 5%
- → A/B test vs current model
- judge on watch time, buys,
- 7-day retention
- auto-rollback on bad latency
Deep dive: cold start
Deep dive five: cold start. A new user has no history, so start with popular items in their region, any interests chosen at sign-up, and context like device; after two or three clicks, session features take over. A new item has no clicks, so build its embedding from content, title, description and image, which places it near similar items in the index. Reserve a few exploration slots to show new items and learn quickly. For users with sparse history, lean on item-to-item lists.
- new user: popular in region,
- onboarding picks, device and
- source of sign-up
- → session signals after 2 clicks
- new item: content embedding
- from title, text, image
- → near similar items in ANN
- reserve exploration slots:
- boost new items, measure fast
- sparse users: lean on item-item
Deep dive: feedback loop
Deep dive six: the feedback loop. The model only learns from what it chose to show, so popular items get shown, clicked and shown more, and niche items never get a chance. Top positions also get clicks just for being on top, so log the position and model that bias. Spend about five percent of slots on exploration using a bandit method. Finally, re-rank for diversity, at most two items per creator, and filter items the user has already seen with a Bloom filter.
- popular gets shown → clicked
- → more popular: rich get richer
- position bias: top slots get
- clicks just for being on top
- → log position, model it
- explore ~5% of slots
- (Thompson sampling, bandits)
- re-rank for diversity: max 2
- per creator, mix categories
- filter seen items (Bloom filter)
Trade-offs
Review the trade-offs. Precomputing recommendations nightly for every user is cheap but stale, and wasted on users who never come back. Running the whole funnel live is fresh but costs real compute. A hybrid caches candidates for an hour and ranks live. If the ranker is slow, skip it and serve candidates in source order; if everything fails, serve a cached popular list. Monitor latency, empty rows, click-through drift and feature freshness.
- precompute nightly per user:
- cheap, stale, wasted on idle
- real-time funnel: fresh,
- ~17k req/s of model compute
- hybrid: cache candidates 1 h,
- rank live
- ranker slow → skip it, serve
- candidate order; all down →
- popular list from cache
- monitor: p99, empty rows, CTR
- drift, feature freshness
Interview summary
In the interview: a funnel from fifty million items to a thousand candidates, five hundred ranked and twenty shown. Candidates from embeddings with nearest-neighbour search, item-to-item, trending and recent. A ranker with rich features predicting several outcomes. A feature store so training matches serving. Train daily, prove it online. Handle cold start with content and popularity, and correct the feedback loop with exploration and diversity.
- funnel: 50M → 1,000 → 500 → 20
- candidates: two-tower + ANN,
- item-item, trending, recent
- ranker: rich features, multi-
- outcome model, blended score
- feature store: one definition,
- log features at serve time
- train daily, prove it online
- cold start: content + popular
- explore, debias, diversify