Notes
Listen
Listen first. Teams create feature flags that are on, off or rolled out to a percentage, with targeting rules like country, app version or internal staff. Experiments have two or more variants with a traffic split. We need gradual ramps, an instant kill switch, and a results page showing lift and confidence for each metric. Every change is audited. Building the product analytics suite itself is out of scope.
The qualities matter more than the features. A flag check sits on every request, so it must take microseconds with no network call. A user must see the same variant on every server and every device. If our platform goes down, product apps must keep running on the last known config. A kill switch must take effect in seconds. Results can be daily, but safety guardrails need to react within minutes.
- feature flags: on, off, or a
- percentage, per environment
- targeting: country, app version,
- plan, internal staff
- experiments: 2+ variants with
- traffic splits
- ramp 1% → 50%, kill instantly
- results: lift and confidence
- per metric, per variant
- audit log of every change
- evaluation: < 1 ms, in process
- no network call per check
- sticky: same user, same variant
- on every server and device
- platform down → apps still run
- on last config or defaults
- kill switch live in < 10 s
- results: correct, not fast;
- daily is fine, guardrails
- within minutes
Ask scale
Scale. Fifty million daily users and a hundred thousand requests a second at peak, each checking about twenty flags, is two million evaluations a second. Done in memory at a microsecond each, that is about two CPU cores across the whole fleet. Exposure events, logged once per user per test per day, come to four hundred million a day, about forty-six hundred a second. The whole config is only four megabytes.
One more number, because interviewers love it. To detect a two percent relative lift on a five percent conversion rate, about a tenth of a point, the rule of thumb is sixteen times p times one minus p, divided by the effect squared. That is about seven hundred sixty thousand users per arm. Even with ten percent of traffic we get that in a day, but we still run a full week so weekday and weekend behaviour both count.
50M daily users, 100k req/s peak 20 flag checks per request = 2M evaluations/s, all local 1 µs each ≈ 2 CPU cores fleet-wide exposures: 50M users × 8 tests = 400M/day ≈ 4,600/s, peak 14k/s × 200 B = 80 GB/day outcome events: 2B/day ≈ 23k/s config: 2,000 flags × 2 KB = 4 MB
baseline conversion p = 5% detect +2% relative = 0.1 pt n per arm ≈ 16 × p(1-p) / δ² = 16 × 0.0475 / 0.001² ≈ 760,000 users per arm 10% of 50M daily = 5M users/day → enough in a day, but run ≥ 1 full week (weekday cycles) tiny lifts need huge samples
Define API
Two kinds of API. Humans use a management API to create experiments with variants, a split, targeting rules, metrics and a layer, and to patch a flag's rollout. Servers fetch the whole config snapshot with an ETag, so unchanged config costs nothing. Product code never calls us over the network for a decision: it calls the SDK in process, and the SDK sends exposure events in batches.
POST /v1/experiments
{ key, variants, split,
targeting, metrics, layer }
PATCH /v1/flags/{key}
{ rollout: 25, enabled }
GET /v1/config?env=prod
→ snapshot, ETag: v4812
SDK: variant("checkout_v2", user)
POST /v1/exposures (batched) Data model
The config is small and relational: flags with rules, a rollout percentage, a salt and a version; experiments with a layer, status, variants and split; and an audit table of every change, with who made it. Events are the opposite: huge, append-only, written once. Exposures and outcome events flow through Kafka into a columnar warehouse, partitioned by day.
flags(key, env, enabled, rules, rollout_pct, salt, version) experiments(key, layer, status, variants, split, start, end) layers(id, salt, bucket ranges) audit(change_id, who, diff, at) exposures(user, exp, variant, ts) events(user, name, value, ts) → Kafka, then warehouse
High-level design
Draw the config path first. A team changes a rollout in the console. After validation and, for production, an approval, the change is written to the config store as a new version. A publisher builds a full snapshot for each environment and pushes it to a CDN or object storage. Every SDK holds that snapshot in memory and refreshes it. The config store is never on the request path.
Now the request path. A user opens checkout. The SDK hashes the user into variant B in memory and the page renders. In the background, the SDK records an exposure event and ships it in a batch to a collector, which writes to Kafka. Events land in the warehouse, a nightly pipeline joins exposures with outcomes like purchases, and the stats engine computes lift per variant.
Deep dive: bucketing
Deep dive one: assignment. We never store who is in which variant. Hash the experiment's salt together with the user id, take it modulo ten thousand, and map bucket ranges to variants. Any server, any phone, computes the same answer with no lookup, and the salt makes each experiment's split independent of the others. For logged-out users, hash a device id, and decide up front whether the test is per device or per account.
Two tests on the same checkout button would contaminate each other. So group experiments into layers. A layer has one salt, and tests inside it get disjoint bucket ranges, so a user is in at most one of them. Tests in different layers overlap freely. When ramping from one to fifty percent, only add buckets, so users already in the test keep their variant. Changing the split of a running test means a new salt and a fresh start.
h = hash(salt + ":" + user_id) bucket = h mod 10,000 checkout_v2, 20% of traffic: 0-999 → A (control) 1000-1999 → B 2000+ → not in test same input → same bucket on every server, no storage logged-out: device id, then merge
- layer = one salt, 10,000 buckets
- tests in one layer get disjoint
- ranges: never overlap
- tests in different layers: any
- user can be in both
- ramp 1% → 5% → 25% → 50%:
- only add buckets, never move
- existing users across arms
- change the split mid-test?
- new salt, restart the clock
Deep dive: delivery
Deep dive two: delivery. Servers poll the CDN snapshot every thirty seconds; an unchanged ETag costs almost nothing. For speed, we also push change notices over a streaming connection, so a kill switch lands in seconds. Mobile apps should not download every rule, which would leak unreleased features, so they fetch evaluated variants once per session. If everything is down, the SDK uses the last snapshot on disk, then the default written in code.
- servers: poll snapshot every
- 30 s with ETag (304 = free)
- + a streaming push for kills
- mobile: ask once per session
- for evaluated variants only
- (do not ship hidden rules)
- platform down: keep last
- snapshot on disk, else the
- default written in code
- kill switch: < 10 s end to end
Deep dive: exposures
Deep dive three: exposures. Log an exposure only when the variant actually changes what the user sees, not when they are bucketed. If we counted everyone who was merely assigned, people who never reached checkout would dilute the result. The SDK deduplicates to one event per user per test per day and batches them. The collector validates, stamps server time, and writes to Kafka keyed by user id, so each user's events stay in order.
- log when the variant is used,
- not when the user is bucketed
- (else: diluted, noisy results)
- SDK dedupes: one per user per
- test per day, in an LRU
- batch 100 events or 5 s
- collector: validate, add server
- time, write to Kafka
- key = user_id: one user's
- events stay in order
Deep dive: metrics
Deep dive four: the metrics pipeline. For each user and test, find the first exposure. Join only the outcome events that happened after it, then roll up per user: purchases, revenue, sessions. Then roll up per variant into a count, a sum and a sum of squares. Those three numbers give the mean and variance, which is all a t-test and a confidence interval need. A daily Spark job over the warehouse does this comfortably.
1 first exposure per (user, test)
2 join outcome events after it
3 roll up per user:
purchases, revenue, sessions
4 roll up per variant:
n, sum, sum of squares
5 mean, variance → t-test, 95% CI
per-user rows: 5M × 300 tests
is fine for a Spark job Deep dive: trust
Before showing any result, check the split. If we planned fifty-fifty but observed fifty point nine to forty-nine point one across two million users, a chi-square test says that is almost impossible by chance, so logging or assignment is broken and the result cannot be trusted. Also stop teams from peeking and ending a test the moment it looks significant: use a fixed length or sequential testing. Name one primary metric up front.
- sample ratio mismatch (SRM):
- planned 50/50, got 50.9/49.1
- on 2M users → chi-square
- p < 0.001 → result is broken
- peeking: stopping when p dips
- under 0.05 inflates errors
- → fixed length or sequential
- many metrics: name 1 primary,
- correct for the rest
- novelty: watch week 2 vs week 1
Deep dive: guardrails
Daily results are too slow to catch a variant that crashes the app. A streaming guardrail job computes error rate, p ninety-nine latency and crash rate per variant every minute. If a variant is three times worse than control for five minutes, it sets the rollout to zero through the normal config path and pages the owning team. Finally, give every flag an owner and an expiry date, because forgotten flags pile up as hidden complexity.
- stream per-variant error rate,
- p99 latency, crash rate
- in 1-minute windows
- variant 3× worse than control
- for 5 min → auto rollback
- + page the owning team
- ramps pause until a human
- looks
- every flag has an owner and
- an expiry: clean up old flags
Trade-offs
Review the trade-offs. Evaluating locally makes checks fast and keeps apps working when we are down, but different servers can run config that is up to thirty seconds apart, which experiments tolerate. Evaluating on the client works offline but exposes rules. Batch results are cheap and correct but a day behind, which is why guardrails stream. Monitor config age per SDK, dropped events, and how often tests fail the sample ratio check.
- local eval: fast, survives our
- outage; but config is up to
- 30 s stale on some servers
- server vs client evaluation:
- hide rules vs work offline
- batch results: cheap, correct
- but a day behind
- Kafka down → SDK buffers,
- then drops (count the drops)
- monitor: config age, SRM rate
Interview summary
The thirty-second version. Assign by hashing a salt and the user id into ten thousand buckets, so it is sticky with no storage. Layers keep conflicting tests apart, and ramps only add buckets. Config is a snapshot on a CDN, held in memory by every SDK, with streaming for kill switches. Log exposures, not assignments, join them to outcomes daily, check the sample ratio, forbid peeking, and stream guardrails.
- hash(salt + user) mod 10,000
- → sticky, no storage
- layers keep tests apart
- ramps only add buckets
- snapshot on CDN, SDK in memory,
- stream for kill switches
- log exposure, not assignment
- join first exposure → outcomes
- SRM check, no peeking
- streaming guardrails