Alert Fatigue

Alert fatigue is the state in which on-call engineers receive so many alerts, many of them noisy, redundant, or non-actionable, that they become numb to them. Pages blur into background noise, and responders start dismissing or muting alerts reflexively. Inevitably, a genuinely critical alert gets ignored along with the noise.

It is the paradoxical failure mode of over-alerting: more alerts producing less safety.

Why It Happens

Alerts accumulate over time, one added after each incident as a "never again" reflex, with none ever removed. Teams alert on causes (a CPU spike, a single failed request) rather than user-facing symptoms, generating many pages that do not actually indicate a problem. Thresholds are set conservatively, producing false positives. Flapping conditions page repeatedly. The signal-to-noise ratio degrades until the whole system is mostly noise.

Why It Hurts

When most alerts are noise, responders learn that pages are usually safe to ignore, which is rational and dangerous. The one alert that matters drowns in the flood and gets the same dismissive treatment, delaying response to real incidents. Constant paging, especially overnight, burns people out and drives attrition. The alerting system, meant to increase reliability, ends up undermining it.

Warning Signs

• On-call is paged constantly, including for non-issues.
• Alerts are routinely acknowledged and ignored without action.
• Many alerts are not actionable when they fire.
• A high fraction of alerts are false positives.

Better Alternatives

Alert on symptoms that affect users, not on every internal cause. Tie alerts to service-level objectives so you page when user experience is actually at risk. Make every alert actionable: if there is nothing to do, it should not page. Continuously tune and prune alerts, removing the ones that never lead to action. Use severity tiers so only true emergencies wake someone up.

How to Refactor Out of It

Audit recent alerts and classify each as actionable or noise. Delete or downgrade the non-actionable ones, and consolidate redundant ones. Re-anchor critical alerts on user-facing SLOs rather than internal causes. Route lower-severity signals to dashboards or tickets instead of pages. Review alert effectiveness regularly, treating a noisy alert as a defect to be fixed, so the signal stays trustworthy.