Thundering Herd
A thundering herd is many actors triggered at the same instant by a shared clock or event, creating a load spike that overwhelms a resource and can stop a service from recovering. Break the synchronization with jitter, coalescing, rate limiting, and staggered restarts.
The thundering herd is a class of problem where a large number of waiting actors are released or triggered simultaneously and all act at once, producing a sharp, correlated load spike. The name comes from the OS case where many threads blocked on one event are all woken when it fires, even though only one can make progress. The same dynamic appears across distributed systems: synchronized cron jobs, caches that expire together, clients that all reconnect when a server restarts, or retries aligned to the same clock.
Why It Happens
Synchronization sneaks in through shared clocks and shared events. Cron jobs scheduled "on the hour" all fire together. Tokens or cache entries created in a batch share an expiry. When a service comes back after an outage, every client that was waiting reconnects in the same second. Each individual behavior is reasonable; the problem is that they are correlated, and nothing spreads them out. It is invisible at low scale and only emerges when enough actors align.
Why It Hurts
A resource sized for average load is hit with a massive instantaneous peak — every client at once — which can exceed capacity and cause failures, even though total daily load is modest. After an outage, the reconnect surge can immediately knock the recovering service back down, preventing it from ever stabilizing (a recovery loop). The spikes also force over-provisioning to survive peaks that last milliseconds, wasting money the rest of the time.
Warning Signs
- Cron schedules or TTLs aligned to round numbers (top of the hour, midnight).
- Mass simultaneous reconnects after a deploy or outage.
- Load graphs with sharp periodic spikes against a low baseline.
- A recovering service repeatedly falling over the moment it comes back.
Better Alternatives
- Jitter / randomization: spread timers, TTLs, retries, and reconnects over a window instead of a single instant.
- Request coalescing / single-flight: collapse many identical concurrent requests into one.
- Rate limiting and token buckets: cap the instantaneous rate so the herd is metered in.
- Staggered / queued startup: ramp clients back gradually after recovery (backoff with jitter, connection limits).
- Wake-one semantics at the OS/lock level so only the thread that can proceed is woken.
How to Refactor Out of It
Find the source of synchronization. Add jitter wherever actors share a clock or event: randomize cron start times, scatter cache TTLs, and use backoff-with-jitter for reconnects and retries so clients return gradually rather than all at once. Put rate limiting in front of the resource so a herd is shaped into a manageable stream. For duplicate concurrent work, coalesce it into a single operation. After a recovery, ramp clients back with connection caps and staggered re-entry. Validate by simulating a mass-reconnect or synchronized-expiry event and confirming the peak is flattened.