Design a News Aggregator: Ingesting Feeds, Clustering Stories and Ranking Fresh News

Listen

Scope it. We ingest articles from publisher feeds, group articles about the same event into one story, and rank stories per section and region, like top stories, business or sports in India. Readers can follow topics and hide sources. Tapping a story opens the publisher's page, so we never host full articles. Non-functional: a new article appears within five minutes, the feed loads in under two hundred milliseconds, and it must stay up during big news. Comments and paywalls are out of scope.

Ask scale

The numbers are friendly. Fifty thousand feeds polled every five minutes is about a hundred and seventy polls a second, producing two million articles a day, roughly twenty-five a second. Fifty million daily readers loading the feed ten times is five thousand requests a second, and during a big event that can jump five times. Storage is about twenty gigabytes a day. So this is a read-heavy system with a small, steady write stream.

ingest: 50,000 feeds polled every 5 min
  50,000 / 300 s = about 170 polls/s
  2M new articles/day = about 25/s

read: 50M DAU x 10 feed loads
  500M / 100,000 s = 5,000 rps
  breaking news peak x5 = 25,000 rps

storage: 2M x 10 KB = 20 GB/day
  = 7 TB/year of metadata + text

Define API

Four endpoints. Get the feed for a section and region with cursor pagination, returning stories with a headline, how many sources covered it, and the lead article. Get one story with all its coverage. Update preferences. And post click events, which feed back into ranking. Make the cursor point at a specific ranking snapshot, otherwise a re-rank between page one and page two shows the reader duplicates.

GET  /v1/feed?section=top&region=in
       &cursor=abc&limit=20
  -> [{ story_id, headline, sources: 37,
        top_article: { url, publisher } }]
GET  /v1/stories/{id}  all coverage
PUT  /v1/me/prefs { follow: [...],
       hide_sources: [...] }
POST /v1/events { story_id, type: click }

cursor = opaque (rank snapshot, offset)

Data model

Sources hold the feed URL, how often to poll it, and an authority score. Articles hold the title, snippet, publish time, extracted entities and an embedding vector, plus the story they belong to. Stories hold a headline, a lead article and a score. A document store or search engine fits articles well, since they are written once and queried by time and text. The ranked lists themselves are just arrays of story ids in Redis, one per section and region.

sources(id, feed_url, interval, etag,
  authority, last_seen)
articles(id, source_id, canonical_url,
  title, snippet, published_at, lang,
  entities[], embedding, story_id)
stories(id, headline, lead_article,
  article_count, first_seen, score)

Redis: feed:top:in -> [story ids]
prefs(user_id, topics[], hidden[])

High-level design

The write path. Pollers fetch publisher feeds and push new articles onto a Kafka stream. An enrichment worker deduplicates, extracts entities, computes an embedding and either attaches the article to an existing story or starts a new one. Every minute, a ranker rescores the active stories and writes a fresh ranked list for each section and region into Redis.

The read path. A logged-out reader asking for top stories gets a list that is identical for everyone in that region, so the CDN serves it with a sixty second lifetime. A logged-in reader hits the feed service, which reads their profile, fetches a few precomputed lists from Redis, merges them and returns twenty stories. No database query sits on the hot read path.

Deep dive: ingestion

Polling everything every five minutes wastes work on quiet blogs and is too slow for wire services. Give each source its own interval: shorten it whenever a poll finds new items, lengthen it when it does not. Use conditional requests so most polls return a tiny 304. Where publishers support a push protocol such as WebSub, subscribe and skip polling. Before anything else, drop exact duplicates by canonical URL, since syndicated articles appear in many feeds.

adaptive interval per source:
  wire service: every 1 min
  weekly blog: every 6 hours
  new item seen -> shorten, else lengthen
conditional GET: ETag / If-Modified-Since
  most polls return a tiny 304
push (WebSub) where supported
partition sources by hash across pollers

dedup: canonical URL, then content hash

Deep dive: clustering

Clustering is the signature problem. Turn each article's title and snippet into an embedding vector and extract named entities. Search a vector index of stories active in the last two days for the nearest ones. If the similarity passes a threshold and they share key entities, the article joins that story; otherwise it starts a new one. This runs incrementally, one article at a time, so it keeps up with the stream. A slower nightly job repairs clusters that should be merged or split.

Putting a Kafka log between ingestion and enrichment buys more than buffering. Keep one to two weeks of raw articles. When you improve the embedding model or the clustering rules, replay the log into a fresh story index, compare quality side by side, then switch. If an enrichment worker crashes, it resumes from its last committed offset. Partition by source id so each publisher's articles stay in order.

1. embed title + snippet (vector)
2. extract entities: people, places
3. candidates: stories active in last
   48 h, same language (vector index)
4. similarity above 0.8 and shared
   entities -> join that story
   else -> start a new story

headline = most central article
nightly: merge or split clusters

Deep dive: ranking

A story's score combines how many distinct sources cover it on a log scale, the authority of those sources, how fast new articles are arriving, and a freshness decay, here halving every six hours. Engagement helps, but cap it, or clickbait wins. Each minute the ranker takes the active stories for each section and region, applies a diversity rule so one topic cannot fill the top ten, and writes a new versioned list.

coverage:  distinct sources, log scale
authority: weighted by source quality
velocity:  new articles in last hour
freshness: x 0.5 ^ (age_hours / 6)
engagement: click rate, capped

every minute, per section and region:
  take top 500 active stories
  diversity: max 2 per topic in top 10
  write list v1903, keep v1902

Deep dive: personalize

Here is the key design choice. Rebuilding fifty million personal feeds every minute would be enormous work for little gain, because news is mostly shared. Instead precompute a few hundred section and region lists, and personalize at read time: fetch the three to five lists a user follows, interleave them by score, remove hidden sources and duplicate stories, and cache that merged page briefly. It is a pull model, which suits content that everyone shares.

per-user feeds: 50M lists, rebuilt
  every minute = far too much work

instead: about 300 sections x regions
  each list precomputed by the ranker

on request: read 3-5 lists the user
  follows, interleave by score,
  drop hidden sources, dedup stories
  cache the merged page for 60 s

Deep dive: breaking news

Breaking news means everyone opens the app in the same minute. Public lists sit on the CDN with a short lifetime and stale while revalidate, so the origin sees a trickle. The feed service is stateless and autoscales. The Redis lists are tiny and replicated. Push alerts go through a separate notification system. And if things get tight, degrade gracefully: skip personalization and serve the regional top list, which is still a good feed.

Deep dive: regions

To serve the world, split write and read. Ingestion and clustering run once, centrally, so there is one consistent set of stories. The ranked lists are only a few megabytes in total, so replicate them to every region. Each region runs its own feed service, Redis and CDN, so readers never cross an ocean. Region is already part of every list key. If one region fails, DNS sends readers to the next closest one.

Trade-offs

Name the trade-offs. Faster polling means fresher news but more load on publishers. The clustering threshold is a dial: too loose and unrelated stories merge, too strict and one event shows up as five stories. Ranking by engagement alone drifts toward clickbait, so authority and diversity are deliberate counterweights. Failures degrade softly: a dead feed just goes stale, and if the ranker stops, readers get the last list, a minute or two old.

The metric that defines this product is the lag from an article's publish time to its first appearance in a feed, with a target like ninety-five percent within five minutes. Also watch poll errors per source, a sampled check of how many stories each real event became, and feed latency and CDN hit rate. Ten times more sources means more pollers partitioned by source. Ten times more readers is mostly CDN and Redis replicas. New languages get their own clustering and ranking.

Interview summary

In the interview: a small steady write stream and a spiky read load. Adaptive polling with conditional requests and push where available. A Kafka stream into enrichment, where embeddings and entities cluster articles into stories. A score built from coverage, authority, velocity and decay, with a diversity rule. Precomputed lists per section and region, blended per user at read time, with the CDN absorbing spikes.