Symptom-Based Alerting

Best Practice: Symptom-Based Alerting

Symptom-based alerting means paging on what the user actually experiences, such as elevated error rate, high latency, or fast SLO error-budget burn, rather than on internal causes like a single full disk or one restarted process. Cause-based alerts multiply with system complexity and produce noise; symptom-based alerts stay small in number and high in signal because they map to real impact.

It matters because alert fatigue is a leading cause of missed incidents and on-call burnout. When every page corresponds to user pain, responders trust the alert and act. The cause is then found through dashboards and traces, not through dozens of competing pages.

The principle behind symptom-based alerting is that you cannot enumerate every way a system can fail, but you can detect that it is failing from the user's point of view. A full disk, a crashed process, or a slow dependency only matter if they harm users; if redundancy absorbs them, no one needs to be paged. The SRE distinction is useful here: page on what is urgent and actionable, send a ticket for what is important but not urgent, and put everything else on a dashboard. Multi-window, multi-burn-rate alerting, described in the SRE Workbook, ties pages directly to SLO error-budget consumption, so a fast burn pages immediately while a slow burn opens a ticket. This keeps the page count low and the signal high.

Step-by-Step Implementation Guidance

1. Define SLOs so you have a precise definition of "the user is being harmed."
2. Create alerts on symptoms: error ratio, latency percentiles, and SLO burn rate.
3. Use multi-window, multi-burn-rate alerting to balance fast detection against false positives.
4. Make cause-based signals dashboards and warnings, not pages.
5. Add for durations so brief blips do not page.
6. Ensure every page is actionable and links to a runbook.
7. Review every page in retrospective: if it was not actionable, tune or delete it.

Common Mistakes Teams Make When Ignoring This Practice

• Paging on CPU, memory, or disk without user impact.
• One alert per failure mode, producing alert storms during incidents.
• No for duration, so transient spikes wake people up.
• Alerts with no runbook or owner.
• Never pruning noisy alerts, eroding trust.

Tools and Techniques That Support This Practice

• Prometheus Alertmanager with multi-burn-rate rules.
• Grafana alerting and SLO burn panels.
• PagerDuty and Opsgenie for routing and escalation.
• OpenSLO-derived burn-rate alert generation.
• Routing rules and on-call schedules to send each page to the right responder.
• Dashboards and traces for cause analysis once a symptom alert fires.

How This Practice Applies to Different Migration Types

• Cloud Migration: Alert on user-facing error and latency during cutover, not on instance churn that autoscaling and replacement handle automatically.
• Database Migration: Page on elevated query errors and timeouts, not on every replica blip.
• SaaS Migration: Alert when end-to-end success drops, regardless of which vendor component failed.
• Codebase Migration: Keep symptom alerts stable so a rewrite that regresses users pages immediately.

Checklist

• SLOs define user harm precisely
• Alerts target symptoms, not causes
• Multi-burn-rate alerting configured
• Cause signals are dashboards, not pages
• Every page links to a runbook
• Noisy alerts reviewed and pruned