Load Shedding

A system pushed past its capacity does not fail gracefully on its own — latency climbs, queues grow, memory fills, and eventually it collapses entirely, often taking longer to recover than the overload lasted. Load Shedding is the deliberate choice to reject some work early so the system can keep serving the rest. Rather than accepting every request and degrading for everyone, the server sheds excess load and stays healthy for the requests it does admit.

How It Works

The server continuously estimates its own health using signals such as queue depth, in-flight request count, CPU utilization, event-loop lag, or measured latency against a target. When a signal crosses a threshold, the server stops admitting new work:

• Threshold shedding: reject requests once concurrency or queue length exceeds a limit, typically returning HTTP 503.
• Priority-aware shedding: drop low-priority traffic (background jobs, analytics, free-tier requests) first, preserving critical paths like checkout and auth.
• Adaptive/feedback control: continuously adjust the admission limit based on observed latency, as in Netflix's concurrency-limits or Google's adaptive approaches.

The rejection must be cheap — a shed request should cost far less than a served one, or shedding cannot help. It is also better to fail fast than to enqueue work that will time out anyway.

When to Use It

Use load shedding for any service that can receive more traffic than it can handle and cannot scale instantly. It is critical for systems with hard latency SLAs, for protecting against traffic spikes and retry storms, and wherever some requests are clearly more important than others.

Trade-offs

Shedding sacrifices some requests on purpose, which is a hard product decision and must be communicated to clients (so they back off rather than hammer harder). Tuning thresholds is difficult: too aggressive wastes capacity, too lenient lets the system tip over. Effective shedding also requires a way to prioritize traffic, which adds complexity.

Related Patterns

Load shedding sits beside rate-limiter (per-client limits vs. system-wide protection), backpressure (signaling upstream to slow down), bulkhead-pattern (containing where overload lands), and graceful-degradation (reducing functionality instead of failing).

Example

Concurrency-based shedding:

max_in_flight = 200
if in_flight.count() >= max_in_flight:
  return 503 with Retry-After
else:
  in_flight.add(); serve(); in_flight.remove()

Envoy's overload manager, the Netflix concurrency-limits library, and Linkerd implement adaptive variants that learn the right limit from latency feedback.