Error Budgets

Best Practice: Error Budgets

An error budget is the maximum acceptable amount of unreliability over an SLO window, calculated as 100% minus the SLO target. For a 99.9% availability SLO over 30 days, the budget is roughly 43 minutes of downtime. The budget reframes reliability from "never fail" to "fail no more than this much," which gives teams a shared, data-driven mechanism for deciding when to ship and when to stop.

Error budgets matter because they convert reliability arguments into accounting. When the budget is healthy, teams can take risks and release quickly. When it is exhausted, the policy shifts the team to reliability work until the budget recovers. This removes emotion and politics from the release decision: the data decides, and everyone agreed to the rule in advance.

The budget also reframes failure as a resource to spend wisely rather than a catastrophe to avoid at all costs. Planned risk, such as a large migration or a risky launch, can be funded from the budget deliberately. A burn-rate alert measures how fast you are consuming the budget; a high burn rate over a short window signals an active incident, while a slow burn over a long window signals a chronic problem. Combining several windows catches both without excessive noise.

Step-by-Step Implementation Guidance

1. Establish SLOs first; the error budget is meaningless without them.
2. Compute the budget for the window: budget = (1 - SLO) x total valid events (or time).
3. Track budget consumption continuously, ideally with a burn-rate alert (e.g. alert if the budget would be spent in under an hour).
4. Write an error-budget policy agreed by engineering and product that says what happens when the budget is low or empty.
5. Make budget status visible on a shared dashboard.
6. Enforce the policy consistently: feature freezes when exhausted, faster releases when healthy.
7. Review burn after every major incident and feed lessons back into the policy.

Common Mistakes Teams Make When Ignoring This Practice

• Treating any outage as equally unacceptable, paralyzing releases.
• Defining a budget but never enforcing the policy.
• Using only slow, monthly windows so fast burn goes undetected.
• Letting management override the policy ad hoc, destroying its credibility.
• Spending the budget on toil rather than learning.

Tools and Techniques That Support This Practice

• Prometheus multi-window, multi-burn-rate alerting (per the SRE Workbook).
• Grafana, Datadog, and Nobl9 error-budget dashboards.
• Sloth and Pyrra to generate budget burn alerts from SLO specs.
• OpenSLO for portable budget definitions.

How This Practice Applies to Different Migration Types

• Cloud Migration: Reserve a portion of the budget explicitly for cutover risk and pause if early burn is steep.
• Database Migration: Track correctness-SLO burn during dual-write to catch divergence before it harms users.
• SaaS Migration: Compare vendor-caused burn against your budget to decide whether the platform meets your needs.
• Codebase Migration: Use a healthy budget as the green light to roll out a rewrite incrementally.

Checklist

• SLOs defined as a prerequisite
• Budget computed for each SLO window
• Burn-rate alerts configured
• Written error-budget policy agreed with product
• Budget status on a shared dashboard
• Policy enforced consistently
• Post-incident burn review in place