codequest / system design
Core concept 11 of 21

Rate limiting

Cap how many requests a client can make so one user or bot cannot take the system down for everyone.

Why limit

Without limits, one buggy client, scraper or attacker can use up all your capacity and take the service down for everyone. A rate limiter caps requests per client (per user, API key or IP) and rejects the excess withHTTP 429 Too Many Requests, usually with a Retry-After header.

Token bucket

Each client has a bucket of tokens that refills at a steady rate up to a maximum. Each request spends a token; no token, no entry. The refill rate is the sustained limit, and the bucket size is how big a burst you allow.

10 / 10 tokens
accepted: 0rejected (429): 0last 10 s

Traffic below the refill rate always passes. A burst passes until the bucket is empty, then requests get 429 until tokens refill. Size the bucket for the bursts you want to allow and the refill for the average you can afford.

Other algorithms

  • Leaky bucket: requests drain from a queue at a fixed rate. Smooth output, but bursts wait or get dropped.
  • Fixed window: count requests per calendar minute. Simple, but a client can send double the limit across a window boundary.
  • Sliding window log or counter: counts over the last 60 seconds, not the calendar minute. Accurate, at the cost of more memory or a weighted estimate.

Where it runs

  • At the edge or API gateway, so rejected traffic never reaches your servers.
  • With many gateway servers, counters live in a shared store such as Redis (atomic increment with expiry) so a client can't spread requests to dodge the limit.
  • Different limits per plan or endpoint: login attempts deserve much stricter limits than reading a feed.

In the interview

  • Say what you key on (user ID or API key; IP only for anonymous traffic) and what the limits are.
  • Pick token bucket when bursts are fine, sliding window when the limit must be exact.
  • Explain distributed counting (Redis) and that clients get 429 plus Retry-After.