Retries, timeouts & circuit breakers
Survive slow or failing dependencies without making the outage worse.
Failures spread
In a system of many services, one slow dependency can take everything down. Requests to it hang, threads and connections pile up waiting, and soon the healthy service calling it runs out of capacity too. Three tools stop the spread.
1. Timeouts
Never wait forever. Every network call needs a timeout a bit above that dependency's normal p99 latency. A request that's going to fail should fail quickly and free its resources.
2. Retries with backoff and jitter
- Retry only errors that might succeed next time (timeouts, 503s), and only idempotent operations.
- Exponential backoff: wait 100 ms, then 200, 400, 800… so a struggling service gets room to recover.
- Jitter: add randomness to each wait so thousands of clients don't retry in sync.
- Cap the attempts. Unlimited retries turn a small outage into a retry storm that multiplies load just when the service is weakest.
3. Circuit breakers
A circuit breaker watches calls to a dependency. After too many failures it opens and fails every call instantly for a cooldown period, instead of making users wait on timeouts. Then it goes half-open: one trial call goes through. Success closes the breaker; failure opens it again.
While the breaker is open, serve a fallback: cached data, a default, or a friendly "try again later", instead of an error page.
Related ideas
- Bulkheads: give each dependency its own connection pool, so one slow service can't use up every thread.
- Load shedding: when overloaded, reject some requests early (503) to keep the rest fast.
In the interview
- When the design calls another service, mention timeouts, retries with backoff and jitter, and a circuit breaker.
- Say what the fallback is when the dependency is down.
- Retrying only works if the operation is idempotent.