How Things Work

Rate Limiting & Throttling

How systems cap request rates to stay up under abuse and overload — token bucket vs leaky bucket vs sliding window, the 429 response, and distributed enforcement with Redis.

How It Works

Rate limiting caps how many requests a client may make in a time window, rejecting the excess with HTTP 429. It protects systems from abuse (brute force, scraping), accidental overload (a buggy client in a loop), and noisy neighbors, while keeping capacity fair across users. The algorithm you pick decides how bursts are handled.

Cap the Request Rate

A rate limiter enforces 'at most N requests per window' for a given identity (user, API key, IP). Requests under the limit pass; requests over it get rejected with HTTP 429 (Too Many Requests), usually with a Retry-After header.

Token Bucket

A bucket holds up to N tokens and refills at a steady rate. Each request consumes one token; if the bucket is empty, the request is rejected. This elegantly allows short bursts (spend the whole bucket at once) while bounding the long-run average to the refill rate.

Window Algorithms

Fixed-window counts requests per clock interval (simple, but allows 2× bursts at window edges). Sliding-window log/counter smooths that boundary by weighting the previous window. Leaky bucket drains at a constant rate, shaping bursty input into smooth output.

Distributed Enforcement

With many API servers, the limit must be shared, so counters live in a central store like Redis and are updated atomically (a Lua script or INCR). The limiter sits at the edge — API gateway or middleware — so abusive traffic is rejected before it reaches your application.

Key Concepts

🪣Token Bucket

A bucket of N tokens refilled at a fixed rate; each request spends one. Allows bursts up to N, average bounded by the refill rate.

💧Leaky Bucket

Requests queue and drain at a constant rate — smooths bursts into a steady output stream (traffic shaping).

🪟Fixed Window

Count requests per clock interval (e.g. per minute). Simple but allows a 2× burst across the window boundary.

🎚️Sliding Window

Weights the current and previous window to avoid the edge-burst problem of fixed windows. More accurate.

🚦429 + Retry-After

The standard 'rate limited' response. Retry-After tells well-behaved clients when to try again.

🐢Throttling

Slowing or rejecting requests above a limit to protect a system from overload or abuse, per identity.

Distributed token bucket (Redis + Lua)

tsx

1-- Distributed token bucket in Redis (atomic via Lua).
2-- One bucket per client key; refills lazily based on elapsed time.
3local key      = KEYS[1]
4local capacity = tonumber(ARGV[1])     -- max burst
5local rate     = tonumber(ARGV[2])     -- tokens per second
6local now      = tonumber(ARGV[3])
7
8local b = redis.call("HMGET", key, "tokens", "ts")
9local tokens = tonumber(b[1]) or capacity
10local ts     = tonumber(b[2]) or now
11
12-- refill for the time that has passed since last request
13tokens = math.min(capacity, tokens + (now - ts) * rate)
14
15local allowed = tokens >= 1
16if allowed then tokens = tokens - 1 end
17
18redis.call("HMSET", key, "tokens", tokens, "ts", now)
19redis.call("EXPIRE", key, math.ceil(capacity / rate) * 2)
20return allowed and 1 or 0   -- 1 = serve, 0 = return HTTP 429

💡

Why This Matters

Every public API needs rate limiting for security, stability, and fairness. It's a common interview topic because it touches algorithms (token/leaky bucket, sliding window), distributed state (atomic counters in Redis), and placement (gateway vs app). Done right, it's the difference between graceful degradation and a cascading outage under attack.

Common Pitfalls

⚠Fixed-window limiters allow a 2× burst at the boundary — switch to sliding window for strict limits.

⚠In-memory counters per instance let clients exceed the global limit by hitting different servers — use shared state.

⚠Non-atomic read-modify-write of the counter causes race conditions under load — use Lua/INCR.

⚠Returning 429 without a Retry-After header — clients then hammer you with blind retries.

⚠Limiting only by IP — NAT/proxies make many users share one IP; combine with user/API-key identity.

Real-World Use Cases

1Stopping Credential-Stuffing on Login

Scenario

Attackers hammer the /login endpoint with millions of username/password guesses from a botnet, threatening account takeover and overloading the auth service.

Problem

Unlimited login attempts let attackers brute-force credentials and also act as a denial-of-service against legitimate users. The app server happily processes every attempt.

Solution

Apply a strict per-account and per-IP rate limit (e.g. 5 attempts/minute with exponential backoff) at the gateway, plus a global limit on the endpoint. Return 429 with Retry-After and trigger CAPTCHA or temporary lockout after repeated denials.

💡

Takeaway: Rate limiting is a frontline security control, not just a performance knob. Limiting sensitive endpoints per identity blunts brute-force and abuse before requests ever reach business logic.

2Protecting a Fragile Third-Party API

Scenario

Your service calls a partner API that allows 100 requests/second. During traffic spikes you blow past it and the partner starts returning 429s, breaking your whole integration.

Problem

Without self-imposed limiting, your outbound calls are as bursty as your inbound traffic, so you exceed the partner's quota and get throttled at the worst possible moment.

Solution

Put a client-side token-bucket (or leaky-bucket) limiter in front of the partner client to shape your outbound rate to just under their quota, and queue overflow instead of dropping it. The leaky bucket converts your bursty calls into a smooth ≤100 r/s stream.

💡

Takeaway: Rate limiting works in both directions — limit what you accept AND what you emit. Shaping outbound traffic keeps you a good citizen of upstream APIs and prevents cascading 429s.

Join the discussion

Comments

Loading comments...

Cookie preferences

We use essential cookies for sign-in and preferences. With your permission, we also use basic analytics to improve lessons.