Notes
The problem
Without a gateway, clients call each service directly. They hard-code addresses, so moving a service breaks the app. Every team re-implements token checks, CORS and rate limiting, each slightly differently. A mobile screen makes six calls over a slow network. And internal endpoints end up on the public internet. The gateway pulls all of that to one place.
- clients know every service
- address (and break on change)
- each service re-does auth,
- CORS, rate limits, logging
- mobile makes 6 calls per
- screen over a slow network
- internal APIs exposed
- to the internet
What it does
The gateway is the single public entry point. It routes by path or host, so slash orders goes to the order service. It verifies the token once, enforces quotas, handles TLS, CORS and size limits, and stamps every request with a trace id. What it must not hold is business logic. The moment it knows what a discount is, you've built a new monolith.
- routing: /orders → orders svc
- auth: verify token once
- quotas and rate limits
- TLS, CORS, request size caps
- transform: headers, versions
- aggregate several calls
- observability: logs, metrics,
- trace ids for every call
- NOT business logic
Inside one request
Inside, a gateway is a chain of filters run in order. Terminate TLS. Match the route. Authenticate the token and check its scopes. Apply the rate limit for this key. Transform the request, adding a user id and request id header. Call the upstream with a timeout, retry once only if the method is idempotent, map errors to a consistent shape, then log, count and trace. Cheap checks first, so bad requests die early.
1 TLS terminate, HTTP/2
2 match route: GET /orders/{id}
3 authenticate: verify JWT
4 authorize: scope orders:read?
5 rate limit: key = api key
6 transform: add X-User-Id,
X-Request-Id, strip cookies
7 call upstream (timeout 2 s)
8 retry once if idempotent
9 map errors, add headers
10 log + metrics + trace span Auth at the edge
Authentication is the gateway's most important filter. With JWTs it verifies the signature using public keys it fetched from the identity provider and cached, so there's no network call per request. The cost: you can't instantly revoke a token, so keep them short-lived. Opaque tokens need a lookup, which you'd cache. Either way, the order service still checks that order ninety-one belongs to user forty-two.
- JWT: verify signature with
- cached public keys (JWKS)
- → microseconds, no call out
- but revocation waits for expiry
- (keep tokens short: 5-15 min)
- opaque token: ask the IdP
- (introspection) → revocable,
- adds a hop; cache results
- services still check ownership
Quotas
Quotas are the second classic job. The gateway counts per API key, per user or per IP, with stricter limits on routes like login, and plan-based quotas like sixty a minute for free and a thousand for pro. Because there are many gateway instances, counters live in a shared store like Redis. Over the limit, the client gets 429 with Retry-After. The Rate limiter building block goes deeper.
- per API key, user or IP
- per route (login is stricter)
- per plan: free 60/min,
- pro 1,000/min, monthly quota
- counters in Redis, or local
- buckets synced in the background
- over the limit → 429
- (see the Rate limiter block)
Aggregation
Gateways can also aggregate. The mobile home screen makes one call; the gateway fans out to users, orders and search in parallel and merges the result. On a phone network that saves a couple of round trips, easily a few hundred milliseconds. Set a deadline and return partial data if recommendations are slow. A common variant is backend-for-frontend: a thin gateway per client type, owned by that client's team.
- mobile: 1 round trip on 4G
- instead of 3 (save ~300 ms)
- fan out in parallel, set a
- deadline, return partial data
- if one part is slow
- backend-for-frontend: one
- thin gateway per client type
- (web BFF, mobile BFF)
- owned by that client team
Variants
Don't confuse it with its neighbours. A load balancer spreads traffic over copies of one service. A reverse proxy forwards and caches without knowing about API keys or plans. The gateway handles north-south traffic, from the outside world in, with API concepts like keys and quotas. A service mesh handles east-west traffic between services, with sidecar proxies doing mutual TLS and retries. Many stacks use all of them.
- load balancer: spreads load
- over copies of ONE service
- reverse proxy: forwards and
- caches; no API concepts
- API gateway: north-south
- traffic, API keys, quotas
- service mesh: east-west,
- sidecars, mTLS, retries
Routing releases
Because all traffic passes through it, the gateway is where releases get controlled. A canary sends five percent of requests to version two while you watch error rates, then ramps up. Header-based routing lets testers opt in. Slash v1 and slash v2 can point at different backends. And shadowing copies real traffic to a new version and throws away its answers, so you can test under real load safely.
- canary: 5% of traffic to v2,
- watch errors, then 25, 50, 100
- header routing:
- X-Beta: 1 → new version
- API versions: /v1 and /v2
- to different backends
- shadow: copy traffic to v2,
- discard its responses
Failure handling
The biggest failure mode: one slow upstream drags everything down. If search starts taking nine seconds, its requests pile up and hold the gateway's connections, and suddenly orders are slow too. Defend with per-route timeouts, a circuit breaker that fails fast once errors pass a threshold, and bulkheads: a separate connection pool per upstream, so search can only exhaust its own.
- danger: slow search holds
- gateway threads/connections
- → orders requests queue too
- timeouts per route (2 s)
- circuit breaker: 50% errors
- in 10 s → fail fast for 30 s
- bulkhead: separate connection
- pool per upstream
- retries only if idempotent
Dependencies
The gateway has its own dependencies, so decide each failure in advance. Identity provider down? Cached signing keys keep verification working. Quota store down? Fall back to rough per-instance limits and fail open for normal routes, but fail closed for login and one-time passwords. And the scariest one is a bad config push that breaks every route at once, so roll config out like code: validate, canary, roll back.
- identity provider down:
- JWT keys cached → keep
- verifying (fail safe)
- Redis (quotas) down:
- fail open with local limits
- for normal routes
- fail closed for login / OTP
- bad config push:
- validate, canary, roll back
Scaling
To scale it, keep the data plane stateless: identical nodes behind a layer four balancer, autoscaled on CPU. Routes, keys and plugins live in a separate control plane that pushes config, and nodes keep serving the last good config if the control plane is down. Watch the latency budget: a gateway should add a few milliseconds, so avoid any blocking network call per request beyond the quota check.
- data plane: stateless nodes
- behind an L4 LB, autoscale
- on CPU or connections
- control plane: stores routes,
- keys, plugins; pushes config
- (data plane keeps last good)
- budget: gateway adds ~1-5 ms
- keep plugins cheap, no
- blocking calls per request
Sizing
Quick sizing. Forty thousand requests a second at peak. If one four-core node doing JWT checks and a quota lookup manages about five thousand a second, which you'd benchmark, that's eight nodes, so run eleven with headroom. The quota store sees one operation per request, forty thousand a second, which a single Redis primary handles comfortably, with a replica for failover. Managed gateways charge per request, which gets expensive at this volume.
peak: 40,000 req/s one node (4 vCPU, JWT + quota): ~5,000 req/s (benchmark it) → 8 nodes, run 11 (+30%) Redis quota checks: 40k ops/s → one primary is plenty (~100k simple ops/s), add a replica for failover managed: per-request pricing
Say it like this
In an interview, draw clients, a load balancer, a stateless gateway fleet, then the services. Say what runs there: TLS, token verification, quotas, routing and trace ids. Say what doesn't: business rules and object-level permissions. Then show it isn't a bottleneck or single point of failure: stateless and horizontally scaled, with timeouts, circuit breakers and bulkheads per upstream.
- draw: clients → LB → gateway
- → services
- say what runs there: TLS,
- JWT verify, quotas, routing,
- trace ids
- and what does not: business
- logic, per-object permissions
- failure: timeouts, breakers,
- bulkheads, fail open/closed