How to Create Prometheus Alerting Rules with Alertmanager

What and why

Metrics are only useful if someone is told when they go wrong. Prometheus evaluates alerting rules and forwards firing alerts to Alertmanager, which deduplicates, groups, routes, and silences them before delivering notifications. Good rules and routing keep on-call signal high and noise low.

Prerequisites

• A running Prometheus server with service metrics.
• A notification destination (Slack webhook or SMTP).

Steps

1. Write alerting rules

Create alerts.yml:

groups:
  - name: availability
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_request_duration_seconds_count{status=~"5.."}[5m]))
            /
          sum(rate(http_request_duration_seconds_count[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "5xx error rate above 5% for 10m"

The for clause requires the condition to hold before firing, which suppresses brief spikes.

2. Load rules into Prometheus

Reference the file and enable Alertmanager in prometheus.yml:

rule_files:
  - alerts.yml
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

3. Run Alertmanager

docker run -p 9093:9093 -v $(pwd)/alertmanager.yml:/etc/alertmanager/alertmanager.yml prom/alertmanager

4. Configure routing and receivers

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: slack
receivers:
  - name: slack
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/XXX'
        channel: '#alerts'

Grouping collapses related alerts into one notification; repeat_interval controls reminders.

5. Test and tune the alert

Generate errors to drive the expression over the threshold. After the for window, the alert fires and a Slack message arrives. Add inhibition rules so a page alert suppresses redundant warning alerts, and use silences during planned maintenance.

Verification

• Prometheus Alerts page shows the rule as pending then firing.
• Alertmanager UI shows the grouped alert.
• A notification arrives at the configured receiver.

Next Steps

Add severity-based routing (page to PagerDuty, warning to Slack), define runbook links in annotations, and build multi-burn-rate alerts tied to your SLOs to page only on user-impacting issues.