Latency, throughput & p95
Why averages lie, why latency explodes near full utilization, and how to talk about p95 and p99.
Latency vs throughput
Latency is how long one request takes. Throughput is how many requests per second the system completes. They are linked: as throughput approaches a component's capacity, requests start waiting in line and latency grows.
Why averages lie
If 95 requests take 20 ms and 5 take 2 seconds, the average is about 120 ms, which describes nobody's experience. Percentiles do better: p50 (median) is the typical request, p95 is the slowest 5%, p99 the slowest 1%. Heavy users make many requests per page, so they hit the tail constantly; that's why SLOs are written as "p99 under 300 ms", not as an average.
The utilization cliff
Queueing theory says waiting time grows like utilization ÷ (1 − utilization). At 50% busy a request waits about one service time; at 90% about nine; at 99% about ninety-nine. Real servers with many workers stay flat a bit longer, then fall off the same cliff. That's why you run at 60–70% at peak, not 95%.
Try it
Three app servers at about 80% busy. Remove one at a time and watch p95 climb, then the error rate jump once the tier passes 100%.
- p95 latency (server)
- 76 ms
- Error rate
- 0.0%
- Est. cost / month
- $290
- excl. bandwidth
Primary DB: 1,250 queries/s (1 query per request). How the numbers work
Numbers worth knowing
| Operation | Roughly |
|---|---|
| Read from memory (cache hit, same machine) | ~0.1 µs |
| Round trip within one data center | ~0.5 ms |
| Read from SSD | ~0.1 ms |
| Simple indexed database query | 1–10 ms |
| Round trip between continents | ~150 ms |
In the interview
- State latency targets as percentiles ("p99 under 200 ms"), not averages.
- Size each tier for 60–70% at peak to leave headroom.
- Find the slowest step on the critical path; move work off it with caches or queues.