Hedged Requests

In large systems, the slowest few percent of requests dominate user-perceived latency — a phenomenon Google's Jeff Dean and Luiz Barroso analyzed in "The Tail at Scale." A single request that hits a server doing garbage collection, a cold cache, or a noisy-neighbor spike can be far slower than the median, even when the service is healthy overall. Hedged Requests attack tail latency: if a request has not responded within a short delay, the client sends a second copy to another replica and uses whichever answer comes back first, cancelling the loser.

How It Works

The client issues the primary request and starts a timer set to a high percentile of normal latency (for example p95). If the primary responds before the timer fires, nothing extra happens. If the timer fires first, the client dispatches a hedge to a different replica. The first response to arrive wins; the other is cancelled if possible.

Because the hedge fires only for the slow tail, extra load is small — typically a few percent of requests are ever duplicated. A related variant, tied requests, sends both copies immediately but lets the servers coordinate so that once one starts processing, it cancels the other, trimming the tail further at slightly higher cost.

When to Use It

Hedging fits read-heavy, idempotent operations against replicated backends where tail latency matters: distributed databases, storage systems, search, and key-value lookups. It is most valuable when latency SLAs are tight and the backend has multiple equivalent replicas to choose from. gRPC and several distributed databases (Cassandra, DynamoDB clients, BigTable) support hedging directly.

Trade-offs

Hedging trades extra load for lower latency, so it must be bounded — fire only on the slow tail and cap the number of hedges, or it can amplify load during a broad slowdown exactly when the system is fragile. It generally requires idempotent operations, since the same request may be processed twice; writes need care or coordination. Cancellation is best-effort, so wasted work is possible. Hedging also assumes slowness is server-specific (a replica having a bad moment), not a systemic overload that every replica shares.

Related Patterns

Hedged requests complement request-coalescing (deduplicating identical concurrent requests), timeout-pattern (bounding each attempt), and contrast with retry-with-backoff (which waits for failure rather than acting on slowness). load-shedding guards against hedging amplifying systemic overload.

Example

Client-side hedging:

primary = send(replicaA)
result = await(primary, timeout=p95_latency)
if result.timedOut:
  hedge = send(replicaB)
  result = awaitFirst(primary, hedge)   // first wins
  cancel(other)
return result

Only genuinely slow requests pay the cost of a second call, while the p99 latency drops sharply.