Notes
Listen
Listen. Users sign up with an email and password, log in and out, and can sign out of every device. They stay signed in for thirty days. They can reset a forgotten password, turn on multi-factor with an authenticator app or a passkey, and log in with Google, Apple or their company's single sign-on. Deciding what a user may do, which is authorization, is a separate problem; we only prove who they are.
The qualities. We never store anything that can be turned back into a password quickly, even if the whole database leaks. A stolen session must be revocable within minutes. Every API call checks a token, so that check should take under a millisecond, ideally without a database hop. We must resist credential stuffing and guessing. And availability is critical: if login is down, the whole product is down.
- sign up with email + password
- log in, log out, log out of
- all devices
- stay signed in for 30 days
- forgot password by email
- MFA: authenticator app,
- passkeys, SMS as a fallback
- log in with Google or Apple
- enterprise SSO for work accounts
- out of scope: permissions (authz)
- never store a usable password
- stolen DB must not reveal
- passwords quickly
- revoke a session within minutes
- token check on every API call:
- < 1 ms, no DB hop if possible
- resist credential stuffing
- and brute force
- login p99 < 500 ms
- 99.99%: no login = no product
Ask scale
Scale. A hundred million users, twenty million daily. Most stay signed in, so real logins are about five million a day, sixty a second, six hundred at peak. Here is the interesting number: a good password hash deliberately costs about a hundred milliseconds of CPU, so six hundred logins a second need sixty cores. An attack at ten times that would melt us, so rate limits run before hashing. Token checks are far busier: a hundred fifty thousand a second.
100M users, 20M daily active logins: 5M/day ≈ 60/s, peak 600/s hash cost ≈ 100 ms of one core 600/s × 0.1 s = 60 cores at peak attack ×10 → limit before hashing API calls: 20M × 200 = 4B/day ≈ 46k/s, peak ≈ 150k/s checks refresh: 15 min tokens ≈ 80M/day ≈ 900/s
Define API
The API is small. Sign up and log in with an email and password. Login either returns tokens, a four oh one, or an MFA challenge that the client completes with a code. Refresh swaps a refresh token for a new pair. Logout can end one session or all of them. Forgot and reset handle lost passwords. Every endpoint is rate limited per IP and per account, because each one is an attack surface.
POST /v1/signup { email, password }
POST /v1/login { email, password }
→ 200 tokens | 401 | mfa_required
POST /v1/mfa/verify { challenge, code }
POST /v1/token/refresh
{ refresh_token } → new pair
POST /v1/logout { all_devices? }
POST /v1/password/forgot { email }
POST /v1/password/reset
{ token, new_password } Data model
Data model. Users live in SQL with a unique email, a verified flag, the password hash and the hashing parameters used. MFA secrets are encrypted, recovery codes hashed. Social and SSO logins map a provider and subject to a user. Refresh and reset tokens are stored only as hashes, like passwords, so a database leak does not hand out live sessions. Login events feed risk checks and the audit trail.
users(id, email UNIQUE, verified, pw_hash, pw_params, status) mfa(user_id, type, secret_enc, recovery_hashes) identities(provider, subject, user_id) UNIQUE(provider,subject) refresh_tokens(hash, family_id, user_id, device, expires, used) reset_tokens(hash, user_id, exp) login_events(user, ip, ok, at)
High-level design
Trace a login. The request reaches the auth service, which loads the user's password hash and verifies it. If MFA is on, it returns a challenge first. On success it issues two things: a short access token, valid fifteen minutes and signed with a key from the key manager, and a long-lived random refresh token whose hash is stored in the session store. On the web both go in secure, HTTP-only cookies; on mobile, in the keychain.
Now every other request. The gateway checks the access token's signature with a cached public key, plus its expiry, issuer and audience. That takes microseconds and needs no call to the auth service, which is why a hundred fifty thousand checks a second are cheap. The gateway forwards the verified user id to product services, which simply trust it. The auth service only sees logins and refreshes.
Deep dive: passwords
Deep dive one: passwords. Never encrypt them and never use a fast hash like SHA-256. Use a slow, memory-hard function such as argon2id, tuned to about a hundred milliseconds, with a unique random salt per user, so identical passwords produce different hashes and an attacker must crack each one separately. Store the parameters beside the hash, and when you raise them, rehash at the user's next successful login. Reject passwords found in breach lists.
- argon2id (or bcrypt, scrypt):
- slow and memory-hard on purpose
- unique random salt per user:
- same password, different hash
- optional pepper: secret key in
- the key manager, not the DB
- store params with the hash;
- rehash on login when raised
- check new passwords against
- known breached lists
Deep dive: sessions
Deep dive two, the classic question: sessions or JWTs? An opaque session id is a random key into a store: revoking it is instant, but every request needs a lookup. A signed token carries the user id and expiry, so anyone with the public key can verify it with no lookup, but once issued it stays valid until it expires. The usual answer is both: a short fifteen-minute signed access token, and an opaque refresh token in the database that we can delete at any time.
- opaque session id:
- + revoke = delete one row
- - a store lookup per request
- signed token (JWT):
- + verify locally, no lookup
- - valid until it expires,
- cannot be recalled
- hybrid (our choice):
- 15 min signed access token
- + opaque refresh token in DB
- revocation lag ≤ 15 min
Deep dive: refresh
Refresh tokens are long-lived, so they get extra care. Each refresh returns a new pair and marks the old refresh token as used, within a family that started at login. If a used token ever comes back, two parties hold copies, so one is a thief: revoke the entire family and make the user log in again. Logging out of all devices, or changing the password, deletes every family. Allow a few seconds of grace for genuine network retries.
refresh R1 → issue A2 + R2, mark R1 used (same family) R1 presented again later? someone copied it → revoke the whole family → force login, alert user logout all devices: delete every family for user password change → same small grace window for retries
Deep dive: reset
Deep dive three: forgot password. Always give the same reply, whether or not the email exists, so attackers cannot discover accounts. The link carries thirty-two random bytes; we store only its hash, it expires in thirty minutes and works once. After a successful reset, revoke every session and email the user a notice, in case it was not them. If MFA is on, still ask for it, or the reset link becomes a way around the second factor.
- always reply "if the account
- exists, we sent a link"
- (no user enumeration)
- token: 32 random bytes, store
- only its hash, expires 30 min
- single use; new one voids old
- after reset: revoke all
- sessions, email a notice
- MFA still required after reset
- rate limit per email and IP
Deep dive: MFA
Deep dive four: multi-factor. An authenticator app shares a secret with us at setup; both sides compute a code from that secret and the current thirty-second window, and we accept one step of clock drift but never the same code twice. Passkeys are stronger: a key pair on the device, bound to our domain, so a phishing site cannot use it. SMS is the weakest because of SIM swaps. Give hashed single-use recovery codes, and ask again for risky actions like changing email.
- TOTP app: shared secret,
- code = HMAC(secret, time/30s)
- accept ±1 step, block reuse
- passkeys (WebAuthn): device key
- pair, bound to our domain,
- phishing resistant
- SMS: SIM swap risk, fallback only
- recovery codes: 10, hashed,
- single use
- step-up MFA for risky actions
Deep dive: OAuth and SSO
Deep dive five: social login and SSO, using OpenID Connect. The app sends the user to the provider, gets back a short code, and our server swaps that code for an ID token, using PKCE so a stolen code is useless. Check the state and nonce values, then verify the token's signature, issuer, audience and expiry. Key users by provider and subject, and only link to an existing account when the email is verified. For work accounts, route by email domain to the company's identity provider.
- authorization code flow + PKCE
- check state (CSRF) and nonce
- verify ID token: signature,
- issuer, audience, expiry
- key users by (provider, sub),
- not by email alone
- link to an existing account
- only if email is verified
- SSO: route by email domain
- to the company's IdP
Deep dive: keys
Deep dive six: keys. Sign tokens with an asymmetric key, so only the auth service can create them while every gateway can verify them. The private key stays in a key management service or hardware module. Each token names its key id, and the public keys are published at a well-known endpoint that verifiers cache. To rotate, publish the new key first, start signing with it a little later, and retire the old one once its last tokens expire. Other secrets belong in a vault, never in git.
- sign tokens with asymmetric
- keys (ES256): only auth signs,
- everyone else verifies
- private key in KMS or HSM
- token header kid = key id
- publish public keys (JWKS);
- verifiers cache for 1 hour
- rotate every 90 days: publish
- new key first, sign with it
- later, retire old after 15 min
- app secrets in a vault, not git
Deep dive: attacks
Finally, attacks. Credential stuffing replays leaked passwords from other sites at huge volume. Rate limit per IP, per account and per network range, before the expensive hash runs. Add growing delays and a CAPTCHA rather than a hard lockout, which an attacker could use to lock real users out. A login from a new device or country triggers MFA and an email. Return the same error for a wrong email or wrong password, and audit every sensitive event.
- limit per IP, per account and
- per IP range, before hashing
- growing delays, then CAPTCHA,
- not a hard lockout (DoS risk)
- flag new device or country:
- require MFA, email the user
- same error for bad email and
- bad password
- audit every login, reset and
- MFA change; alert on spikes
Trade-offs
Review the trade-offs. Short signed access tokens make every check free, but a revoked user keeps access for up to fifteen minutes; for urgent cases, push a small denylist of token ids to the gateways. A slow hash protects users after a breach but costs real CPU. If the auth service goes down, existing tokens keep working, which buys time. Monitor login success rate, four oh one spikes, refresh token reuse and reset volume.
- 15 min access token: fast, but
- revocation waits up to 15 min
- (urgent: short denylist)
- slow hash: safer DB, more CPU
- auth service down: existing
- tokens keep working 15 min
- session store down: no refresh,
- replicate it across zones
- monitor: login success rate,
- 401s, refresh reuse, resets
Interview summary
In the interview: hash passwords with argon2id and a salt, and do the CPU math. Issue a short signed access token, verified at the gateway with no lookup, and an opaque, hashed, rotating refresh token whose reuse revokes the family. Resets use random, hashed, single-use links with the same reply every time. Prefer passkeys and authenticator codes over SMS, use OIDC with PKCE for social login, and keep signing keys in a key manager.
- argon2id + salt, ~100 ms
- → 60 cores at 600 logins/s
- access token 15 min, signed,
- verified at the gateway
- refresh token opaque, hashed,
- rotated, reuse kills family
- reset: random, hashed, 30 min,
- one use, same reply always
- MFA: passkeys > TOTP > SMS
- OIDC code + PKCE, KMS keys