Rate Limiter

Any shared service has finite capacity. Without limits, a single aggressive client, a buggy retry loop, or a denial-of-service attack can consume all of it and degrade service for everyone. A Rate Limiter enforces an upper bound on the number of operations a caller may perform per unit of time, rejecting or delaying requests that exceed the allowance. It protects backends, enforces fair usage across tenants, defends against abuse, and caps cost on metered downstream resources.

How It Works

Several algorithms trade accuracy against memory and burst behavior:

• Token bucket: tokens refill at a steady rate; each request consumes one. Allows bursts up to the bucket size while bounding the average rate. The most common choice.
• Leaky bucket: requests enter a queue drained at a fixed rate, smoothing bursts into a steady outflow.
• Fixed window: count requests per calendar window (e.g. per minute). Simple but allows double-rate bursts at window boundaries.
• Sliding window log / counter: track timestamps or weighted counts across a rolling window for smoother, more accurate limiting.

Limiters apply per key — API key, user, IP, or tenant. In distributed deployments, counters live in a shared store such as Redis so all nodes enforce one global limit. Exceeded requests typically return HTTP 429 with a Retry-After header.

When to Use It

Use rate limiting on any public or multi-tenant API, on login and other abuse-prone endpoints, and in front of expensive or third-party dependencies that bill per call. It is essential for tiered/quota-based products and for protecting a service that cannot scale instantly under sudden load.

Trade-offs

Limits that are too strict frustrate legitimate users and can break batch workloads; too loose, they fail to protect. Distributed rate limiting adds a shared-state dependency and latency, and exactly-once global counting is hard — many systems accept slight over-admission for performance. Communicating limits clearly (headers, documented quotas) is essential, or clients cannot adapt.

Related Patterns

Rate limiting overlaps with throttling (often used synonymously, though throttling implies slowing rather than rejecting), and complements load-shedding, bulkhead-pattern, and backpressure, which manage overload from other angles.

Example

A token-bucket limiter at an API gateway:

bucket: capacity=100 tokens, refill=10/sec per API key
on request:
  if bucket.take(1): forward()
  else: respond 429, Retry-After: 1

Envoy, NGINX, Kong, and cloud API gateways provide rate limiting natively, often backed by Redis for cluster-wide enforcement.