SYSTEM DESIGN · DESIGN PROBLEMS · PLATFORM SERVICES · HARD · 42 OF 51

Design an A/B Testing and Feature Flag Platform: Bucketing, Exposures and Kill Switches

Design the platform every product team uses to ship behind flags and run experiments: deterministic hash bucketing with no lookups, layers for overlapping tests, config snapshots pushed to SDKs that evaluate locally, ramping and kill switches in seconds, exposure logging, a metrics pipeline that joins exposures to outcomes, sample size math, sample ratio checks and guardrails.

Try it on the practice board Drag the blocks, wire them up, and check your design against the reference.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
  42. 42
  43. 43
  44. 44
  45. 45
  46. 46
  47. 47
  48. 48
  49. 49
  50. 50
  51. 51
1 / 19

Press play to hear the narration, or step through at your own pace.

Notes

Listen

Listen first. Teams create feature flags that are on, off or rolled out to a percentage, with targeting rules like country, app version or internal staff. Experiments have two or more variants with a traffic split. We need gradual ramps, an instant kill switch, and a results page showing lift and confidence for each metric. Every change is audited. Building the product analytics suite itself is out of scope.

The qualities matter more than the features. A flag check sits on every request, so it must take microseconds with no network call. A user must see the same variant on every server and every device. If our platform goes down, product apps must keep running on the last known config. A kill switch must take effect in seconds. Results can be daily, but safety guardrails need to react within minutes.

Functional
  • feature flags: on, off, or a
  • percentage, per environment
  • targeting: country, app version,
  • plan, internal staff
  • experiments: 2+ variants with
  • traffic splits
  • ramp 1% → 50%, kill instantly
  • results: lift and confidence
  • per metric, per variant
  • audit log of every change
Non-functional
  • evaluation: < 1 ms, in process
  • no network call per check
  • sticky: same user, same variant
  • on every server and device
  • platform down → apps still run
  • on last config or defaults
  • kill switch live in < 10 s
  • results: correct, not fast;
  • daily is fine, guardrails
  • within minutes

Ask scale

Scale. Fifty million daily users and a hundred thousand requests a second at peak, each checking about twenty flags, is two million evaluations a second. Done in memory at a microsecond each, that is about two CPU cores across the whole fleet. Exposure events, logged once per user per test per day, come to four hundred million a day, about forty-six hundred a second. The whole config is only four megabytes.

One more number, because interviewers love it. To detect a two percent relative lift on a five percent conversion rate, about a tenth of a point, the rule of thumb is sixteen times p times one minus p, divided by the effect squared. That is about seven hundred sixty thousand users per arm. Even with ten percent of traffic we get that in a day, but we still run a full week so weekday and weekend behaviour both count.

Back of the envelope
50M daily users, 100k req/s peak
20 flag checks per request
  = 2M evaluations/s, all local
  1 µs each ≈ 2 CPU cores fleet-wide

exposures: 50M users × 8 tests
  = 400M/day ≈ 4,600/s, peak 14k/s
  × 200 B = 80 GB/day
outcome events: 2B/day ≈ 23k/s
config: 2,000 flags × 2 KB = 4 MB
How many users per variant?
baseline conversion p = 5%
detect +2% relative = 0.1 pt
n per arm ≈ 16 × p(1-p) / δ²
  = 16 × 0.0475 / 0.001²
  ≈ 760,000 users per arm

10% of 50M daily = 5M users/day
  → enough in a day, but run
  ≥ 1 full week (weekday cycles)
tiny lifts need huge samples

Define API

Two kinds of API. Humans use a management API to create experiments with variants, a split, targeting rules, metrics and a layer, and to patch a flag's rollout. Servers fetch the whole config snapshot with an ETag, so unchanged config costs nothing. Product code never calls us over the network for a decision: it calls the SDK in process, and the SDK sends exposure events in batches.

API
POST /v1/experiments
  { key, variants, split,
    targeting, metrics, layer }
PATCH /v1/flags/{key}
  { rollout: 25, enabled }
GET  /v1/config?env=prod
  → snapshot, ETag: v4812

SDK: variant("checkout_v2", user)
POST /v1/exposures  (batched)

Data model

The config is small and relational: flags with rules, a rollout percentage, a salt and a version; experiments with a layer, status, variants and split; and an audit table of every change, with who made it. Events are the opposite: huge, append-only, written once. Exposures and outcome events flow through Kafka into a columnar warehouse, partitioned by day.

Data model
flags(key, env, enabled, rules,
  rollout_pct, salt, version)
experiments(key, layer, status,
  variants, split, start, end)
layers(id, salt, bucket ranges)
audit(change_id, who, diff, at)

exposures(user, exp, variant, ts)
events(user, name, value, ts)
  → Kafka, then warehouse

High-level design

Draw the config path first. A team changes a rollout in the console. After validation and, for production, an approval, the change is written to the config store as a new version. A publisher builds a full snapshot for each environment and pushes it to a CDN or object storage. Every SDK holds that snapshot in memory and refreshes it. The config store is never on the request path.

Now the request path. A user opens checkout. The SDK hashes the user into variant B in memory and the page renders. In the background, the SDK records an exposure event and ships it in a batch to a collector, which writes to Kafka. Events land in the warehouse, a nightly pipeline joins exposures with outcomes like purchases, and the stats engine computes lift per variant.

Deep dive: bucketing

Deep dive one: assignment. We never store who is in which variant. Hash the experiment's salt together with the user id, take it modulo ten thousand, and map bucket ranges to variants. Any server, any phone, computes the same answer with no lookup, and the salt makes each experiment's split independent of the others. For logged-out users, hash a device id, and decide up front whether the test is per device or per account.

Two tests on the same checkout button would contaminate each other. So group experiments into layers. A layer has one salt, and tests inside it get disjoint bucket ranges, so a user is in at most one of them. Tests in different layers overlap freely. When ramping from one to fifty percent, only add buckets, so users already in the test keep their variant. Changing the split of a running test means a new salt and a fresh start.

Deterministic bucketing
h = hash(salt + ":" + user_id)
bucket = h mod 10,000

checkout_v2, 20% of traffic:
  0-999     → A (control)
  1000-1999 → B
  2000+     → not in test

same input → same bucket on
  every server, no storage
logged-out: device id, then merge
Layers and ramps
  • layer = one salt, 10,000 buckets
  • tests in one layer get disjoint
  • ranges: never overlap
  • tests in different layers: any
  • user can be in both
  • ramp 1% → 5% → 25% → 50%:
  • only add buckets, never move
  • existing users across arms
  • change the split mid-test?
  • new salt, restart the clock

Deep dive: delivery

Deep dive two: delivery. Servers poll the CDN snapshot every thirty seconds; an unchanged ETag costs almost nothing. For speed, we also push change notices over a streaming connection, so a kill switch lands in seconds. Mobile apps should not download every rule, which would leak unreleased features, so they fetch evaluated variants once per session. If everything is down, the SDK uses the last snapshot on disk, then the default written in code.

Getting config everywhere
  • servers: poll snapshot every
  • 30 s with ETag (304 = free)
  • + a streaming push for kills
  • mobile: ask once per session
  • for evaluated variants only
  • (do not ship hidden rules)
  • platform down: keep last
  • snapshot on disk, else the
  • default written in code
  • kill switch: < 10 s end to end

Deep dive: exposures

Deep dive three: exposures. Log an exposure only when the variant actually changes what the user sees, not when they are bucketed. If we counted everyone who was merely assigned, people who never reached checkout would dilute the result. The SDK deduplicates to one event per user per test per day and batches them. The collector validates, stamps server time, and writes to Kafka keyed by user id, so each user's events stay in order.

Exposure, not assignment
  • log when the variant is used,
  • not when the user is bucketed
  • (else: diluted, noisy results)
  • SDK dedupes: one per user per
  • test per day, in an LRU
  • batch 100 events or 5 s
  • collector: validate, add server
  • time, write to Kafka
  • key = user_id: one user's
  • events stay in order

Deep dive: metrics

Deep dive four: the metrics pipeline. For each user and test, find the first exposure. Join only the outcome events that happened after it, then roll up per user: purchases, revenue, sessions. Then roll up per variant into a count, a sum and a sum of squares. Those three numbers give the mean and variance, which is all a t-test and a confidence interval need. A daily Spark job over the warehouse does this comfortably.

Metrics pipeline
1 first exposure per (user, test)
2 join outcome events after it
3 roll up per user:
    purchases, revenue, sessions
4 roll up per variant:
    n, sum, sum of squares
5 mean, variance → t-test, 95% CI

per-user rows: 5M × 300 tests
  is fine for a Spark job

Deep dive: trust

Before showing any result, check the split. If we planned fifty-fifty but observed fifty point nine to forty-nine point one across two million users, a chi-square test says that is almost impossible by chance, so logging or assignment is broken and the result cannot be trusted. Also stop teams from peeking and ending a test the moment it looks significant: use a fixed length or sequential testing. Name one primary metric up front.

Results you can trust
  • sample ratio mismatch (SRM):
  • planned 50/50, got 50.9/49.1
  • on 2M users → chi-square
  • p < 0.001 → result is broken
  • peeking: stopping when p dips
  • under 0.05 inflates errors
  • → fixed length or sequential
  • many metrics: name 1 primary,
  • correct for the rest
  • novelty: watch week 2 vs week 1

Deep dive: guardrails

Daily results are too slow to catch a variant that crashes the app. A streaming guardrail job computes error rate, p ninety-nine latency and crash rate per variant every minute. If a variant is three times worse than control for five minutes, it sets the rollout to zero through the normal config path and pages the owning team. Finally, give every flag an owner and an expiry date, because forgotten flags pile up as hidden complexity.

Guardrails
  • stream per-variant error rate,
  • p99 latency, crash rate
  • in 1-minute windows
  • variant 3× worse than control
  • for 5 min → auto rollback
  • + page the owning team
  • ramps pause until a human
  • looks
  • every flag has an owner and
  • an expiry: clean up old flags

Trade-offs

Review the trade-offs. Evaluating locally makes checks fast and keeps apps working when we are down, but different servers can run config that is up to thirty seconds apart, which experiments tolerate. Evaluating on the client works offline but exposes rules. Batch results are cheap and correct but a day behind, which is why guardrails stream. Monitor config age per SDK, dropped events, and how often tests fail the sample ratio check.

Trade-offs and failures
  • local eval: fast, survives our
  • outage; but config is up to
  • 30 s stale on some servers
  • server vs client evaluation:
  • hide rules vs work offline
  • batch results: cheap, correct
  • but a day behind
  • Kafka down → SDK buffers,
  • then drops (count the drops)
  • monitor: config age, SRM rate

Interview summary

The thirty-second version. Assign by hashing a salt and the user id into ten thousand buckets, so it is sticky with no storage. Layers keep conflicting tests apart, and ramps only add buckets. Config is a snapshot on a CDN, held in memory by every SDK, with streaming for kill switches. Log exposures, not assignments, join them to outcomes daily, check the sample ratio, forbid peeking, and stream guardrails.

Say it like this
  • hash(salt + user) mod 10,000
  • → sticky, no storage
  • layers keep tests apart
  • ramps only add buckets
  • snapshot on CDN, SDK in memory,
  • stream for kill switches
  • log exposure, not assignment
  • join first exposure → outcomes
  • SRM check, no peeking
  • streaming guardrails

Interview questions

How do you make sure a user always sees the same variant?

Compute the variant from a hash of the experiment salt and a stable user id, mapped into fixed bucket ranges. Every server and device gets the same answer without any stored assignment, as long as the salt and ranges do not change.

Why not call a central service to decide each flag?

Flags are checked many times per request, so a network call would add latency and make every product depend on the platform being up. Shipping a config snapshot to an in-process SDK makes checks take microseconds and keeps apps running during an outage.

How does a kill switch take effect within seconds?

The change is written as a new config version and a notice is pushed over a streaming connection to SDKs, which refetch the snapshot immediately; periodic polling of the CDN with ETags is the fallback if the stream is down.

What is the difference between assignment and exposure?

Assignment is which variant a user would get; exposure is the moment the variant actually changes what they see. Analysing only exposed users avoids diluting the effect with people who never reached the feature.

How do you run several experiments at once without them interfering?

Put experiments that touch the same surface in one layer with disjoint bucket ranges, so a user joins at most one of them, and let experiments in different layers overlap since they use independent salts.

What is a sample ratio mismatch and why does it matter?

It is when the observed split differs from the planned split by more than chance allows, tested with chi-square. It signals broken assignment or logging, so the experiment results should not be trusted until the cause is found.

How long should an experiment run?

Long enough to reach the sample size needed for the smallest effect you care about, roughly 16 times p(1-p) over the effect squared per arm, and at least one full week to cover weekly patterns. Stopping early on a good-looking p-value inflates false positives.

How do you ramp a feature safely?

Increase the rollout in steps such as 1, 5, 25 and 50 percent by adding buckets so existing users keep their variant, watch streaming guardrail metrics at each step, and roll back automatically if errors or latency regress.