Start practising

SYSTEM DESIGN · CONCEPTS · SERVICES AND HOW THEY TALK · EPISODE 15

Stateless vs Stateful Services and Sessions

What state means for a server, why stateless services scale and recover easily, where session state goes instead (sticky sessions, a shared session store, signed tokens), which parts must stay stateful (databases, caches, WebSocket gateways), and how to explain it in an interview.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
1 / 10

Press play to hear the narration, or step through at your own pace. Every picture here is the same one used in the YouTube video.

Notes

The bug

Here's the classic bug. You log in, and server one stores your session in its own memory. Your next request goes through the load balancer to server two, which has never heard of you. You're logged out. Worse, when server one restarts, every session on it disappears.

State in server memory
  • server 1 stores the session
  • in its own memory
  • the load balancer sends the
  • next request to server 2
  • → "please log in again"
  • and if server 1 restarts,
  • every session on it is gone

Definitions

A stateful server remembers things between requests: sessions, carts, open connections. A stateless server keeps nothing between requests: each request carries, or points to, everything needed, so any server can handle it. Stateless servers are interchangeable. You can add, remove or restart them freely.

Stateless vs stateful
  • stateful: the server remembers
  • things between requests
  • (sessions, carts, open sockets)
  • stateless: every request carries
  • or points to what is needed;
  • any server can handle it
  • stateless servers are
  • interchangeable: add, remove,
  • restart freely

Sticky sessions

Option one: sticky sessions. The load balancer always sends a given user to the same server, using a cookie. No code changes, but if that server dies the sessions die with it, load gets uneven, and every deploy logs people out. It's a quick fix, not a design.

Sticky sessions
  • load balancer pins a user to
  • one server (cookie or IP hash)
  • + no code changes
  • − server dies → sessions lost
  • − uneven load
  • − deploys log people out
  • a quick fix, not a design

Session store

Option two: a shared session store. The cookie holds only a random session id, and the session data lives in Redis with an expiry. Any server can look it up, servers stay stateless, and logging someone out is just deleting the key. The cost is one fast extra hop, and Redis itself must be highly available.

Shared session store
cookie holds only a session id
session data lives in Redis:
  session:9f2c → {user: Asha,
                 cart: [7, 12]}
  TTL 30 min

+ any server works
+ revoke by deleting the key
− one more hop (~0.5 ms)
− Redis must be highly available

Tokens

Option three: signed tokens, like JSON web tokens. The token itself holds the user id and an expiry, plus a signature. Any server verifies the signature without a lookup. The catch: you can't easily revoke a token before it expires, so keep access tokens short-lived and keep refresh tokens in a store.

Signed tokens (JWT)
{ sub: "user_42", role: "user",
  exp: 15 minutes }
+ signature

server checks the signature,
no lookup needed

− cannot easily revoke before
  expiry → keep it short-lived,
  refresh tokens in a store

Where state lives

State doesn't disappear; it moves. Web and API servers and background workers become stateless. Databases, caches, queues and WebSocket gateways are stateful by nature. The design goal is to push state down into a few specialised systems that are built for replication and failover.

State has to live somewhere
  • stateless: web/API servers,
  • workers
  • stateful (by design):
  • databases, caches, queues
  • WebSocket gateways
  • (they hold live connections)
  • push state down to a few
  • specialised, replicated systems

Payoff

The payoff is huge. Stateless servers can autoscale in seconds, be replaced one by one during deploys without logging anyone out, and crash without losing anything. The stateful parts scale differently, through replication and sharding, which is what the database episodes cover.

Why it matters
  • autoscaling: add servers in
  • seconds, any can serve
  • deploys: replace servers one
  • by one, nobody logged out
  • failure: a crashed server
  • loses nothing
  • spot / cheap instances usable
  • stateful systems scale
  • with replication + sharding

Say it like this

In an interview, say it in one line: the application servers are stateless, so any server can take any request and we autoscale them freely. Sessions live in Redis behind a session cookie, or in short-lived tokens. All durable state lives in the database, cache and queue.

In the interview
  • "App servers are stateless"
  • sessions in Redis (session id
  • cookie) or short-lived JWTs
  • any server behind the LB
  • autoscale on CPU / RPS
  • state in DB, cache, queue
  • WebSocket gateways: registry
  • (episode 9)

Interview questions

What is the difference between a stateless and a stateful service?

A stateful service keeps data about clients between requests, such as sessions or open connections. A stateless service keeps nothing between requests, so any instance can handle any request.

Why are stateless services easier to scale?

Instances are interchangeable, so you can add or remove them behind a load balancer, autoscale, deploy and restart without losing user data or needing coordination.

What are sticky sessions and what are their drawbacks?

The load balancer routes a user to the same server every time. Drawbacks: sessions are lost when that server dies, load becomes uneven, and deploys or scaling disrupt users.

Where should session data live in a scalable web app?

In a shared, fast store such as Redis keyed by a random session id in a cookie, or encoded in short-lived signed tokens like JWTs.

What are the trade-offs of JWTs versus server-side sessions?

JWTs need no lookup and work across services, but cannot easily be revoked before expiry and grow request size. Server-side sessions are easy to revoke and small, but need a highly available store and an extra lookup.

Which components in a system are necessarily stateful?

Databases, caches, message queues, object storage and connection gateways (WebSockets). They are scaled with replication, partitioning and failover rather than simple cloning.

How do you handle WebSocket connections, which are stateful?

Keep the gateways focused on holding connections, record which gateway each user is on in a registry, route messages via pub-sub, and have clients reconnect and resume after failures.