Retry Storm

A retry storm is what happens when many clients retry failed requests aggressively and in unison during a partial outage. Each retry adds load to a dependency that is already failing because it is overloaded, so retries make the overload worse, which causes more failures, which trigger more retries — a self-amplifying feedback loop that converts a brief degradation into a sustained outage.

Why It Happens

Retries are a standard resilience tactic and usually a good one. The trouble is naive retries: fixed short intervals, unlimited attempts, no jitter, and retries stacked at every layer (client, gateway, service, library) so one user action becomes exponentially many backend calls. When the dependency hiccups, all clients fail at nearly the same instant and retry at nearly the same instant, synchronizing into waves. It works fine for the occasional transient error and only becomes catastrophic during a correlated failure.

Why It Hurts

The storm directs more traffic at the component least able to handle it, removing any chance for it to recover. Synchronized retry waves create thundering-herd spikes. Layered retries multiply: three layers each retrying three times is up to 27x the original load. The dependency stays pinned at the failure threshold, the outage persists far longer than the original cause, and the blast radius spreads to everything sharing that dependency.

Warning Signs

• Retries with fixed, immediate intervals and no cap on attempts.
• Backend traffic that rises when error rates rise.
• Retry logic present at multiple layers of the call stack.
• Outages that persist long after the triggering event, with synchronized request spikes.

Better Alternatives

• Exponential backoff with jitter: spread retries out and randomize them to break synchronization.
• Circuit breakers: stop calling a failing dependency entirely until it recovers, shedding load.
• Retry budgets / token buckets: cap retries as a fraction of total traffic so they can never multiply load.
• Retry only at one layer and only idempotent, retry-safe operations.
• Load shedding and back-pressure on the server to reject excess fast rather than queue it.

How to Refactor Out of It

Audit every retry in the call path and consolidate to a single layer. Replace fixed-interval retries with capped exponential backoff plus full jitter, and bound total attempts. Add a retry budget so retries can never exceed a small fraction of normal traffic. Put a circuit breaker in front of unreliable dependencies so the system stops hammering them and fails fast during an outage. On the server side, add load shedding so it can protect itself. Test by injecting dependency failures under load and confirming traffic stays bounded and the system recovers on its own.