Lesson 10 introduced the token-bucket algorithm on a whiteboard. This lesson takes it all the way to a real distributed system — one that has to agree with itself correctly across many machines, several data centers, and every request in the fleet.
Where to enforce the limit
The API gateway (or a dedicated middleware layer) sees every request before it fans out to any backend service — which makes it the cheapest place to reject an over-limit request. A rejection here costs almost nothing; a rejection buried deep inside a backend service means auth, parsing, and maybe a database call already happened before the request gets turned away.
The Redis-backed token bucket
Each user gets a key, e.g. ratelimit:{user_id}, storing a token count and a last-refill timestamp. On each request, the gateway computes tokens earned since the last refill (capped at bucket capacity), checks if at least one token is available, and if so decrements one and allows the request — otherwise rejects it.
Chosen for atomic, sub-millisecond operations on the hot path
RedisWhy atomicity is the whole problem
A naive implementation — GET the count, check it, then INCR — has a race condition: two concurrent requests from the same user can both read the same count before either updates it, letting MORE than the limit through. The fix is running the whole read-check-decrement sequence as a single Redis Lua script, which Redis executes atomically on its own server — no other command can interleave in the middle of it. This is the exact detail that separates a correct distributed rate limiter from a subtly broken one.
Multiple data centers
If API gateways run in three data centers but Redis lives in only one, every request pays cross-region latency, and that single Redis instance becomes a new single point of failure for the whole system. A common mitigation: give each region its own local Redis with a LOCAL budget (roughly the global limit divided across regions), periodically syncing usage to rebalance — trading perfect global accuracy for low latency and regional independence. There is no free option here; naming this trade-off explicitly is the strong answer.
Graceful responses: 429 and Retry-After
When a client exceeds its limit, the API should return HTTP 429 Too Many Requests, ideally with a Retry-After header telling the client exactly how long to wait. This matters doubly here: a client that retries a rate-limited request instantly defeats the entire purpose of the limiter — a clear signal for how long to back off is what makes well-behaved clients actually behave well.
Common mistakes
- Implementing the check as separate GET-then-INCR calls instead of one atomic script.
- Assuming a single global Redis instance works fine across multiple regions without acknowledging the latency and SPOF cost.
- Returning a generic error instead of 429 with Retry-After, encouraging clients to hammer the limiter immediately again.
- Enforcing only a per-user limit with no global limit, missing the case where many well-behaved users still add up to too much aggregate load.
Quick recap
| Concept | One-liner |
|---|---|
| Enforcement point | The API gateway — reject cheaply, before backend work happens. |
| Redis + Lua script | Makes the check-and-decrement atomic under concurrency. |
| Multi-datacenter | Local budgets per region trade perfect accuracy for low latency. |
| 429 + Retry-After | Tells well-behaved clients exactly how long to back off. |
Practice Zone
Five MCQs, then two applied questions.
In a distributed rate limiter design, why is the API gateway usually the best place to enforce limits, rather than each individual backend service?
Asked in
Why is Redis (rather than the main relational database) the typical choice for storing the shared token-bucket counters in a distributed rate limiter?
Asked in
A rate limiter's Redis instance lives in one datacenter, but the service has API gateways in three datacenters worldwide. What problem does this create, and what's a reasonable mitigation?
Asked in
When a client exceeds its rate limit, what should the API return, and what header helps the client behave well?
Asked in
A rate limiter enforces both a per-user limit (100 req/min) and a global limit across all users (1,000,000 req/min) on the same endpoint. Why have both?
Asked in
Design a Redis-based token-bucket rate limiter for 'max 100 requests per user per minute.' Sketch the key structure and the logic on each incoming request, and explain how you keep the check-and-decrement atomic under concurrent requests from the same user.
Asked in
Your rate limiter needs to work across 3 regions with a single global limit of 10,000 req/min per user, but you want to avoid every request making a cross-region call. Propose an approximate approach and state what you're trading away.
Asked in
FAQ
Why Redis specifically, and not the main relational database?
A rate-limit check happens on every single request, so it must be extremely fast and cheap — Redis's in-memory atomic operations comfortably handle that hot-path load in a way a disk-backed database checked on every request would struggle to match.
Is a perfectly accurate global limit always necessary?
Rarely — most systems accept a slightly approximate global count (via per-region budgets) in exchange for much lower latency and no cross-region single point of failure.
Should the rate limiter itself have a fallback if Redis is down?
Yes — a well-designed limiter typically fails open (allows requests through) or fails closed (rejects them) deliberately, as an explicit decision, rather than the whole gateway crashing when its Redis dependency is unavailable.