Throttling

Throttling limits how much of a system's resources a given consumer can use over time. When demand exceeds what the system can safely serve, the system constrains, queues, or rejects excess requests rather than degrading or failing for everyone.

The problem is that load can spike beyond provisioned capacity, whether from legitimate bursts, a single noisy tenant, or abuse. Without limits, one consumer can exhaust shared resources and cause cascading failures.

How It Works

The system measures usage per consumer, tenant, or operation against a defined limit, such as requests per second, concurrent operations, or compute units. When a consumer crosses the limit, the system responds in one of several ways: reject the request (often with HTTP 429 and a Retry-After header), degrade gracefully by disabling non-essential features, or defer work by enqueuing it.

Throttling can be reactive, triggered when monitored metrics breach a threshold, or proactive, applied continuously per quota. It is frequently enforced at an API gateway or a sidecar so application code stays simple.

When to Use It

Use it to protect services with hard capacity limits, to enforce fair use and contractual quotas across tenants in a multi-tenant system, and to preserve availability and SLAs during spikes. It is essential for public APIs.

Avoid surprising clients: throttling should be predictable, documented, and communicated through clear response codes and headers.

Trade-offs

Throttling protects the system but rejects or delays work, which affects user experience; limits must balance protection against usability. Setting thresholds is hard and often needs tuning with real traffic. Distributed throttling requires shared state (a counter store) to enforce a global limit, which adds latency and a coordination point.

It addresses short-term overload; it is not a substitute for adequate capacity or autoscaling for sustained growth.

Related Patterns

Rate Limiting is a specific throttling technique using algorithms like token bucket. Queue-Based Load Leveling defers excess work instead of rejecting it. Circuit Breaker stops calls to a failing dependency, complementing inbound throttling.

Example

A public REST API allows each API key 100 requests per minute. An API gateway tracks usage in Redis. When a client exceeds the limit, the gateway returns 429 with Retry-After: 30. Premium tenants get a higher quota. During a regional traffic surge, the service also applies reactive throttling: when CPU passes 85%, it temporarily disables an expensive recommendation feature for free-tier users to keep core endpoints responsive.