Token Bucket (Rate Limiting)

The token bucket is a rate-limiting algorithm that controls how frequently an action — typically API requests — may occur, while tolerating short bursts. It is the most widely used throttling mechanism in API gateways and protects services from overload, abuse, and runaway costs. Rate limiting sits at the boundary between performance and security: it defends availability against both accidental floods and deliberate denial-of-service.

How It Works

Imagine a bucket that holds up to B tokens and refills at a steady rate of R tokens per second. Each request removes one token; if the bucket has a token, the request proceeds, otherwise it is rejected (or queued). When idle, the bucket refills up to its capacity B, which is the maximum burst size; the sustained rate over time converges to R. This lets a client that has been quiet briefly exceed the average rate, then settle back — matching real traffic, which is bursty.

It contrasts with the related leaky bucket (which smooths output to a constant rate, no bursts) and simple fixed/sliding windows (which count requests per interval but can allow double-rate spikes at window edges). In distributed systems the bucket state is held in a shared store like Redis, often updated atomically with a script, so all gateway nodes enforce one limit per client.

When to Use It

Use token bucket to rate-limit public and partner APIs, to enforce per-user or per-IP quotas, to protect expensive endpoints (search, report generation, LLM calls), and to shield backends from traffic spikes. It is the default choice when you want a clean average limit but must allow legitimate bursts, which is most real-world API traffic.

Trade-offs

You must tune two parameters (rate and burst) per use case; too tight frustrates legitimate users, too loose fails to protect. Distributed enforcement adds a shared-state dependency and latency, and must handle races atomically. Choosing the limit key (user, IP, API key) affects fairness — IP limits can punish users behind shared NAT. Clients need clear feedback: return 429 Too Many Requests with Retry-After and rate-limit headers so well-behaved clients back off.

Related Patterns

Token bucket is a throttling pattern that complements the circuit breaker (which stops calls to failing dependencies) and the bulkhead (which isolates resource pools). Rate limits are part of a defense-in-depth posture and are often versioned alongside the API contract.

Example

bucket: capacity B=100 tokens, refill R=10 tokens/sec

on request:
  refill bucket by R * (now - last_refill), capped at B
  if tokens >= 1: tokens -= 1; allow
  else: reject with 429 Too Many Requests, Retry-After: ceil((1 - tokens)/R)

A client may burst up to 100 requests after idling, then is held to a sustained 10 requests per second as the bucket refills.