Rate Limiting & API Design Explained

One user, or one buggy script, sends 100,000 requests a second to your API. Your servers could technically survive it — but should you let them try? Rate limiting is how you say no, deliberately and fairly, before abuse (accidental or otherwise) becomes an outage for everyone else.

Fixed window and its boundary problem

The simplest algorithm: allow N requests per fixed time window (e.g. "100 per minute"), resetting the counter sharply at the boundary. It's easy to implement but has a real weakness: a client can send its full quota right at the END of one window (11:59:59) and its full quota again right at the START of the next (12:00:01) — nearly double the intended rate in a short burst around the boundary.

Token bucket

A bucket holds up to a fixed number of tokens, refilled at a steady rate; each request costs one token, and a request with no tokens available is rejected. The key advantage over fixed-window: unused tokens accumulate (up to capacity) during quiet periods, so a legitimate burst — a mobile app syncing after being briefly offline — can be served instantly using saved-up tokens, while the refill rate still caps the long-run average correctly. This matches how real traffic behaves — bursty, not perfectly smooth — far better than a hard fixed window.

Rate limiting across many servers

A naive local counter on each of 10 API servers doesn't work: a load balancer can route one user's requests to any of the 10 servers, so the user could get 10x the intended limit just by spreading requests across the fleet. The fix is a SHARED counter — typically stored in Redis, updated with an atomic increment or a Lua script — so the limit is enforced against the user's TOTAL traffic across the entire fleet, not per-server slices of it.

Idempotency keys

A payment API lets a client retry a request that timed out, in case only the RESPONSE (not the actual charge) was lost. Without protection, that retry could charge the user twice. An idempotency key — generated once per logical action on the client and sent with every attempt, including retries — lets the server recognize "I've already processed this exact request" and simply replay the earlier result instead of repeating the side effect. This is the standard fix (Stripe's API is the textbook example) for making an inherently non-idempotent operation safe to retry.

Pagination and API versioning

For a large, changing dataset, offset-based pagination (OFFSET 10000 LIMIT 20) forces the database to walk past every skipped row, getting slower as the offset grows, and can show duplicates or skip items if rows are inserted or deleted mid-scroll. Cursor-based pagination anchors to a stable position (an id or timestamp) instead, staying fast and correct regardless of concurrent changes. Separately, API versioning (/v1/orders vs /v2/orders) lets you change an API's shape without instantly breaking every external client still depending on the old one.

Common mistakes

• Using fixed-window limits without acknowledging the boundary-burst weakness.
• Enforcing rate limits with per-server local counters in a multi-server fleet.
• Allowing retries on a payment-style endpoint without an idempotency key.
• Choosing offset-based pagination for a large, frequently-changing dataset.

Quick recap

Practice Zone

Five MCQs, then two applied questions.

Q1Rate limiting algorithmsmedium

What is the main weakness of the fixed-window rate limiting algorithm (e.g. '100 requests per minute, resetting every minute')?

Asked in

aIt uses too much memory to trackbA burst of requests right at the boundary between two windows can let through nearly double the intended limit (e.g. 100 at 11:59:59 and 100 more at 12:00:01)cIt cannot be implemented with RedisdIt always under-counts requests

Q2Token bucketmedium

In the token bucket algorithm, what happens when a client sends requests slower than the token refill rate for a while, then suddenly sends a burst?

Asked in

aThe burst is always rejected outright, no exceptionsbUnused tokens accumulate in the bucket (up to its capacity), so the burst can be served immediately using the saved-up tokens, then throttled once the bucket emptiescToken bucket cannot handle bursts at all, only steady ratesdThe refill rate doubles automatically

Q3Distributed rate limitinghard

Why can't each server in a fleet of 10 API servers just keep its own local in-memory counter for a 'per-user, 100 requests/minute' rate limit?

Asked in

aIt works fine — local counters are always correctbA load balancer can route a user's requests to any of the 10 servers, so each server only sees a fraction of that user's traffic — the user could effectively get 1000 requests/minute by spreading requests across all serverscIn-memory counters are too slow to checkdLocal counters use too much disk space

Q4API design — idempotencymedium

A payment API lets a client retry a POST /charge request if it times out (in case the response, not the charge, got lost). What prevents the user from being charged twice?

Asked in

aNothing — this is an unsolved problembAn idempotency key sent by the client — the server stores the result keyed by that id, so a retry with the same key returns the original result instead of charging againcRate limiting alone solves thisdThe client should just never retry

Q5Paginationeasy

Why is cursor-based pagination (e.g. 'give me items after id=8842') usually preferred over offset-based pagination (e.g. 'give me items 100-120') for a large, frequently-changing feed?

Asked in

aCursor pagination is always faster to implementbOffset pagination re-scans and skips rows on every page (getting slower for later pages) and can show duplicates or skip items if rows are inserted/deleted between requests; a cursor anchors to a stable position insteadcOffset pagination cannot return more than 10 itemsdThere is no real difference, it's purely stylistic

FAQ

Next: Lesson 11 — Reliability, Failure Handling & Disaster Recovery →