A program that listens on a network port and answers requests; by extension, the machine (physical, virtual, container or serverless runtime) that runs it.

What happens inside a web server when a request arrives?

The OS hands the connection to the process listening on that port; the server parses the HTTP request, routes it to a handler, the handler does the work (cache, database, other services), then the server writes the status, headers and body back.

Thread-per-request vs event loop?

Thread-per-request gives each request a thread (simple, but memory-heavy and threads mostly wait on I/O). An event loop uses one or a few threads with non-blocking I/O to multiplex many connections (very efficient for I/O-bound work, but CPU-heavy tasks block it).

Stateful vs stateless servers?

A stateful server keeps client data in its own memory, so requests must return to the same server. A stateless server keeps state in a shared store (database, Redis), so any instance can serve any request, which makes horizontal scaling and failover easy.

Vertical vs horizontal scaling?

Vertical: a bigger machine (simple, but limited and a single point of failure). Horizontal: more machines behind a load balancer (needs stateless design, but scales further and tolerates failures).

What Is a Server, Really? | System Design Concepts 4

Notes

Definition

A server is a program that waits for requests and answers them. The word is also used for the machine that runs that program. But the program is the important part.

Try it yourself. Run python3 -m http.server 8000. Open localhost:8000 in your browser. Your laptop is now a web server, serving files from that folder.

Try it

$ python3 -m http.server 8000
Serving HTTP on port 8000 …

open http://localhost:8000

Listening on a port

So what makes a program a server? It listens on a port. It asks the operating system: send me anything that arrives on port 443. The operating system keeps a table of which program owns which port.

When a connection arrives for that port, the operating system hands it to the program. Only one program can listen on a given port at a time, which is why you sometimes see the error: address already in use.

Where servers live

The program can run on a physical machine, on a virtual machine sharing hardware with others, inside a container, or even serverless, where your code runs per request and the provider manages the machines. It's the same idea, packaged differently.

Machine types

physical server (bare metal)
virtual machine (many per machine)
container (lighter, shares OS)
serverless (code runs per request,
you never see the server)

Request lifecycle

Now, what happens inside when a request arrives? First, the server parses the bytes into a request. Then it routes it: which piece of code handles GET /users/42?

Then the handler does the work. Often it checks a fast in-memory cache first. On a miss, it queries the database, which is slower, and then saves the result in the cache for next time.

Finally it builds the response: a status code, headers, and a body, often JSON for an app or HTML for a browser, and sends it back. That's the whole lifecycle: parse, route, work, respond.

Many at once

Here's where it gets interesting. A real server doesn't get one request at a time. Thousands arrive together. How does one program handle them all?

Model one: a thread per request. Each request gets its own thread of execution. The code is simple. But every thread costs memory, and most of them spend their time just waiting on the database or the network. Classic Java servers like Tomcat work this way, with a thread pool.

Model two: an event loop. One thread juggles thousands of connections. It never sits waiting. It starts a database call, moves on to the next request, and comes back when the result is ready. Node.js and Nginx work like this.

The rule of thumb: work that mostly waits, like calling databases and other services, suits an event loop or lightweight threads. Work that mostly thinks, like image processing, needs real parallel workers, roughly one per CPU core.

Model 1: a thread per request

each request gets its own thread
simple code, easy to reason about
but: each thread costs memory
and most of them just wait
(on the database, the network)

Model 2: event loop

one thread, many connections
never waits: starts the I/O,
moves on, comes back when ready
great for lots of slow I/O
used by Node.js and Nginx

Rule of thumb

I/O-bound work → async / event loop
or many lightweight threads
CPU-bound work → real parallelism:
one worker per CPU core

Stateful vs stateless

Next, an idea every system design interview comes back to: stateful versus stateless. A stateful server keeps your data in its own memory, like your shopping cart. That works, until you need more than one server.

If your next request lands on a different server, your cart is gone. That's why we prefer stateless servers.

A stateless server keeps nothing important in its own memory. Sessions and carts live in a shared store, like a database or a cache such as Redis. Now any server can handle any request, and you can add or remove servers freely.

Scaling

That leads to scaling. Option one: scale up, also called vertical scaling. Buy a bigger machine: more CPU, more memory. It's simple, but there's a ceiling, it gets expensive, and it's still one point of failure.

Option two: scale out, also called horizontal scaling. Run many copies of a stateless server, and put a load balancer in front to spread the requests. If one dies, the others carry on. This is how big systems grow, and it's the next episode.