Notes
Listen
Listen. We check every text post, comment, image, video and profile. Actions are graded: allow, add a label, limit reach, age-restrict, remove, or suspend the account. Policies differ by country and by surface, since ads are stricter than private messages. Unclear cases go to trained human reviewers. Authors can appeal, and other users can report content. Writing the policies themselves is out of scope; we build the machinery that applies them.
The qualities. Checks that run while the user posts may add at most about a hundred fifty milliseconds. Known illegal media must be blocked before anyone can see it. Everything else can be decided asynchronously, most of it within a minute. Items that are spreading fast are reviewed first, because harm grows with views. And every decision must be explainable and reversible, since appeals and regulators will ask.
- check text, images, video,
- live comments, profiles
- actions: allow, label, limit
- reach, age-gate, remove,
- suspend the account
- policies differ by region
- and by surface
- human review for unclear
- appeals by the author
- reports from other users
- inline budget: ≤ 150 ms added
- to posting text or an image
- known illegal media: blocked
- before it is ever visible
- async checks: most decided
- within a minute
- high-reach items reviewed
- first
- every decision explainable
- and reversible
Ask scale
Scale. Five hundred million items a day: four hundred million texts, about forty-six hundred a second; ninety million images, a thousand a second; and ten million videos of about a minute, which at one frame per second is sixty-nine hundred frames a second. At peak, about twenty-four thousand image inferences a second. But look at people: sending half a percent to humans is two and a half million reviews a day, over two thousand full shifts. Getting to a tenth of a percent saves eighty percent of that.
500M items/day
text 400M ≈ 4,600/s
images 90M ≈ 1,000/s
video 10M × 60 s, 1 frame/s
≈ 6,900 frames/s
peak ×3 ≈ 24,000 media inf/s
to humans at 0.5% = 2.5M/day
20 s each → 2,300 shifts/day
at 0.1% → about 460 shifts
→ routing precision is money Define API
The API. Product services call an inline check while the user posts, and get allow, hold or block within the budget. Every created item also emits an event for the deeper asynchronous scan, and the verdict comes back as a moderated event that the product applies. Reviewers fetch the next task from a queue and submit a decision with the policy it falls under. Authors file appeals.
POST /internal/moderation/check
{ content_id, type, text | url,
author, surface, region }
→ { action: allow | hold | block }
event content.created → async scan
event content.moderated
{ content_id, action, policy }
GET /v1/review/next?queue=hate-es
POST /v1/review/{task}/decision
POST /v1/appeals { content_id } Data model
The data model separates signals from decisions. Signals are scores per model version, like ninety-one percent likely hateful. Decisions record who decided, a model, a reviewer or an appeal, which policy applied and which action, and they are append-only, so a reversal is a new decision, not an edit. Review tasks carry a queue, a priority, a due time and a lease. Hash lists hold fingerprints of known violating media.
content_state(content_id, status,
visibility, version)
signals(content_id, model, ver,
scores {hate: .91, spam: .02})
decisions(id, content_id, source
auto|human|appeal, policy,
action, actor, at) append-only
review_tasks(id, content_id, queue,
priority, due, lease_until)
hash_list(phash, list, severity) High-level design
The inline path. The upload goes to object storage, and the content API runs only cheap, decisive checks: a fingerprint lookup against known-bad media, account rules like a brand-new account posting fifty links, and a tiny text model. If any is clearly bad, block. Otherwise publish, perhaps with limited reach for risky accounts, and emit an event. This keeps posting fast for the honest majority.
The asynchronous path. Classifiers consume the event and produce scores. The policy engine turns scores into an action for this surface and region: very confident violations are removed automatically, clear passes are allowed, and the uncertain middle goes to a human review queue. Every outcome, automatic or human, is written to the decision log, and the product applies it.
Deep dive: cascade
Deep dive one: the cascade. Start with an exact hash, which catches re-uploads of files already removed. Then a perceptual hash, which still matches an image after resizing or cropping. Then rules: posting velocity, link domains, account age. Then a small, fast model on every item, and only the five percent or so it is unsure about go to the large, expensive model. Each stage is cheaper than the next, which cuts the inference bill by around ten times.
- 1 exact hash: re-upload of
- removed file → block
- 2 perceptual hash: same image
- resized, cropped → block
- 3 rules: velocity, links,
- new account, bad domains
- 4 small fast model: all items
- 5 big model: only the ~5%
- the small one is unsure of
- → GPU bill cut by ~10x
Deep dive: policy
Deep dive two: policy. Models only output scores; the policy engine turns them into actions, with thresholds per policy, surface and region kept as configuration, not code. For violence in the feed, above ninety-seven percent remove, eighty to ninety-seven goes to people, sixty to eighty limits reach, and below is allowed. Ads use stricter numbers. A new rule first runs in shadow mode, logging what it would have done, so you can see its effect before it touches anyone.
policy: violence, surface: feed score ≥ 0.97 → remove, notify 0.80 - 0.97 → human review 0.60 - 0.80 → limit reach < 0.60 → allow surface: ads → stricter numbers region DE → extra legal rules new rule: run in shadow mode, log what it would do, then on
Deep dive: video
Video is the expensive case. Split long videos into chunks and scan them in parallel. Sample a frame a second plus every scene change, turn the audio into a transcript for the text models, and check the title and thumbnail first, since they are what people see in feeds. Keep the video visible only to its author until the first pass finishes, then widen it. Live streams are sampled every few seconds, and a strong hit can cut the stream.
- split into 1-minute chunks,
- scan chunks in parallel
- frames: 1 per second plus
- every scene change
- audio → transcript → text
- models; title, thumbnail too
- visible to the author only
- until the first pass ends
- live streams: sample every
- few s, cut the stream on hit
Deep dive: review
Deep dive three: human review. Order tasks by expected harm: severity, times how fast the item is spreading, times how unsure the model is. Split queues by policy and language, so a Spanish hate speech case reaches a trained Spanish reviewer. Each task is leased for ten minutes and returns to the queue if abandoned. Quality comes from hidden test items with known answers and two reviewers on hard cases. Protect reviewers too: blur graphic media by default and cap their exposure.
priority = severity
× reach (views/hour)
× model uncertainty
queues by policy + language
lease task 10 min, else requeue
SLA: child safety 1 h,
spam 24 h
quality: hidden gold tasks,
2 reviewers on hard cases
wellbeing: blur, grayscale, caps Deep dive: appeals
Appeals. The author sees which policy was applied and can appeal. The appeal goes to its own queue and to a different, usually more senior, reviewer, who sees the full decision history. If it is overturned, a new decision restores the content and its reach, and the original reviewer or model gets that feedback. Because decisions are append-only, the whole story stays visible for audits and transparency reports.
Deep dive: feedback
Deep dive four: the feedback loop. Reviewer and appeal decisions become labels, and models are retrained regularly. But there is a trap: you only get labels for what the model already flagged, so you never learn what it misses. Send a small random sample of automatically allowed items to review as well, which measures recall honestly. New models run in shadow first, then ramp gradually. When new fingerprints are added to hash lists, rescan older content.
- labels from review + appeals
- → training set, weekly
- trap: you only see what the
- model flagged, not misses
- fix: send a random sample of
- auto-allowed items to review
- → measures recall honestly
- new model: shadow, compare,
- then ramp 1% → 100%
- rescan old items on new hashes
Trade-offs
Trade-offs. Precision against recall is tuned per policy: for child safety we accept more false alarms, for mild spam we accept more misses. If classifiers fail, decide per surface: hold media from new accounts, but let trusted users' text through and rescan later. If the review backlog grows, raise automation only for low-harm policies. Re-check items when their views cross thresholds, and expect adversaries to adapt with misspellings and edited images, so retrain often.
The key metric is prevalence: of every ten thousand views, how many were of violating content, measured by sampling. Removals alone can rise while users see more harm. Also track precision and recall per policy, the appeal overturn rate, queue age against each deadline, inline latency, and cost per item. Growing ten times means more languages and regions, and distilling big models into smaller, cheaper ones.
- precision vs recall: tune per
- policy, not one global bar
- classifier down: new accounts
- and media → hold (closed);
- trusted text → allow, rescan
- review backlog → raise auto
- thresholds only for spam
- viral item: re-check when
- views cross 10k, 100k
- adversaries adapt: retrain fast
- prevalence: violating views
- per 10k views (sampled)
- precision and recall by policy
- appeal overturn rate
- queue age vs SLA per queue
- inline check p99 latency
- GPU utilization, cost per item
- 10x: more regions, more
- languages, smaller models
Interview summary
In the interview: a tight inline budget for hashes, rules and a tiny model, with everything else asynchronous. A cascade from hashes to rules to small then large models, so expensive inference only runs where it matters. A policy engine with thresholds per policy, surface and region. Review queues ordered by severity, reach and uncertainty, routed by language. Appeals to a different reviewer. And labels plus random samples feeding retraining with shadow testing.
- inline ≤ 150 ms: hashes, rules,
- tiny model; else async
- cascade: hash → rules → small
- → large model
- policy engine: thresholds per
- policy, surface, region
- queues: severity × reach ×
- uncertainty, by language
- appeals to a new reviewer
- labels + random samples →
- retrain, shadow, ramp