← All posts
general

Design a Rate Limiter: System Design Walkthrough

"Design a rate limiter" looks like a small question and is anything but: in forty minutes it touches algorithms, distributed state, failure philosophy, and API design. It's also beloved by interviewers because every layer has a genuine trade-off — there is no single right answer, only answers you can or cannot defend.

Requirements first

  • Limit requests per client (API key, user ID, or IP) — e.g. 100 requests/minute.
  • Work across a fleet: the client talks to many servers; the limit is global.
  • Add minimal latency (<1–2ms) to every request.
  • Tell rejected clients what happened (HTTP 429, Retry-After) rather than silently dropping.
  • Decide behaviour when the limiter itself is unhealthy — this one question separates levels, so flag it early and return to it.

The algorithms

Walk the options; land on a recommendation.

  • Fixed window. Count per (client, minute), reject over the cap. Trivially cheap — and famously wrong at the edges: 100 requests at 11:59:59 plus 100 at 12:00:01 is 200 in two seconds, all "legal".
  • Sliding window log. Store every request timestamp, count the last 60s. Exact, but memory scales with traffic — the thing you're rate limiting. Mention, discard.
  • Sliding window counter. Weighted blend of the current and previous fixed windows. Smooths the boundary burst at near-fixed-window cost; the pragmatic default at scale.
  • Token bucket. Tokens drip in at rate R up to burst capacity B; each request spends one. Two numbers by design — steady rate and burst allowance — which is why it's the classic choice for public APIs, and stored as just (tokens, last_refill) per client with lazy refill on access.

Recommend token bucket for its explicit burst semantics, with sliding-window counter as the alternative when strict windows are the product requirement.

Making it distributed

Local in-process counters fail the moment a load balancer spreads one client across servers. The standard answer is centralised counters in Redis:

  • State per client is tiny (two numbers), reads/writes are sub-millisecond, and TTLs garbage-collect idle clients for free.
  • The race: read-modify-write from concurrent servers double-spends tokens. Fix it atomically — a Lua script (or INCR-based scheme) so check-and-decrement is one operation. Naming this race unprompted is the difference between having seen the problem and having read about it.
  • Scale-out: shard by client key; a hot client is still one shard, which is fine — that's exactly the client you're limiting.
  • Latency shave: run the limiter check in the API gateway/middleware, one Redis round-trip per request; batch or locally pre-approve for ultra-hot paths if the interviewer pushes.

When the limiter breaks

The philosophical probe: Redis is down — do you reject everything (fail-closed) or wave everything through (fail-open)?

For most products, fail-open with alarms: the rate limiter protects quality of service, and taking the whole API down to enforce quotas inverts the priority. Fail-closed belongs where the limiter is a security control — login attempts, OTP guesses, password resets. Give the two-sided answer with examples; a degraded middle option (fall back to per-instance local limits) earns extra credit.

Telling the client

A limiter without communication just looks like an outage:

  • 429 Too Many Requests with Retry-After, plus X-RateLimit-Limit / -Remaining / -Reset on every response so well-behaved clients can pace themselves before hitting the wall.
  • Document the limits publicly; surprise limits generate support tickets, not compliance.
  • Distinguish per-second burst limits from daily quotas — different windows, different headers, different messaging.

How this plays in an interview

A strong 40 minutes: requirements incl. the failure question (5) → algorithms compared, token bucket recommended (10) → Redis + atomicity + sharding (15) → fail-open/closed reasoning (5) → client-facing headers (5). The trap to avoid is jumping straight to "Redis INCR" without the algorithm comparison — the algorithms are the demonstration that you design rather than pattern-match.

Warm up with the URL shortener, then take on payments or the full WhatsApp deep dive. For live reps against a real interviewer, get matched with a coach.