codequest / system design
Core concept 1 of 21

Horizontal scaling & load balancing

Add identical stateless servers behind a load balancer instead of buying one bigger machine.

Up or out

When one server can't keep up you have two options. Vertical scaling buys a bigger machine: simple, but it has a ceiling, gets expensive fast, and is still one machine that can fail. Horizontal scaling adds more identical machines and spreads traffic across them. That has no hard ceiling and survives a machine dying, which is why almost every large system scales out.

Horizontal scaling only works if servers are stateless: any server can handle any request because session data lives somewhere shared (a cache or database), not in one server's memory.

The load balancer

A load balancer sits in front of the servers and decides which one gets each request. Common strategies:

  • Round robin: take turns. Simple and fine when requests cost about the same.
  • Least connections: send to the server with the fewest requests in flight. Better when some requests are slow.
  • Hash-based (for example by user ID): the same user lands on the same server, useful for local caches. See consistent hashing.

It also runs health checks and stops sending traffic to servers that fail them. Layer 4 balancers route by IP and port; layer 7 balancers understand HTTP and can route by path or header.

Try it

Two app servers are getting 2,000 requests per second. Each handles about 400. Add servers until it is healthy, and notice the latency once you are well below full.

App servers overloaded
p95 latency (server)
281 ms
Error rate
60.0%
Est. cost / month
$250
excl. bandwidth
App servers
400 req/s each · $40/mo
2
Hit rate
DB read replicas
+1,250 reads/s each · also applies every write · $120/mo
0

Primary DB: 1,250 queries/s (1 query per request). How the numbers work

Aim for roughly 60–70% utilization at peak. Latency stays flat until about 80% busy and then climbs steeply, so headroom is what absorbs a spike or a server dying.

In the interview

  • Say early that the app tier is stateless and scaled horizontally behind a load balancer.
  • Size it: peak requests per second ÷ requests one server handles ÷ target utilization.
  • Mention health checks and that the load balancer itself needs redundancy (a pair, or a managed service).
  • Don't stop there: scaling the app tier usually moves the bottleneck to the database.