Toil Reduction Best Practices

Best Practice: Toil Reduction

In Site Reliability Engineering, toil is operational work that is manual, repetitive, automatable, tactical, and scales linearly with the size of the service, while adding no enduring value. Examples include manually restarting services, hand-running the same recovery steps, or processing routine requests. Toil reduction is the deliberate practice of measuring this work and engineering it away.

It matters because unmanaged toil grows with the system and crowds out the engineering that makes services better. Google SRE recommends capping toil (often around 50% of time) so teams retain capacity for automation and reliability work. Reducing toil improves morale, lowers error rates, and frees engineers to handle the change load of a migration.

Not all operational work is toil. Reading a postmortem, designing a system, or making a judgement call is overhead or genuine engineering, not toil. Toil specifically is the work that is manual, repetitive, automatable, devoid of lasting value, and that grows at least linearly with the service. The danger of toil is that it is insidious: each task feels small and reasonable in the moment, so it is rarely questioned, yet in aggregate it can consume a team and leave no time to fix the underlying causes. Measuring toil makes it visible and therefore budgetable, which is the first step to controlling it. The goal is not zero toil, which is rarely worth the cost, but keeping it below a ceiling so that the team always retains capacity to improve the system rather than merely run it.

Step-by-Step Implementation Guidance

Define toil clearly for your team and track time spent on it.
Identify the largest sources of toil by frequency and effort.
Set a target ceiling for toil as a fraction of engineering time.
Prioritize automating the highest-toil tasks first, often via runbook automation.
Eliminate root causes where possible (fix the bug that requires the restart).
Protect engineering time so automation work actually happens.
Re-measure toil periodically to confirm it is trending down.

Common Mistakes Teams Make When Ignoring This Practice

Accepting repetitive manual work as just "the job."
Never measuring toil, so it stays invisible and unbudgeted.
Automating low-frequency tasks while ignoring the big ones.
Treating symptoms (restart scripts) instead of root causes.
Letting toil consume all time, starving reliability work.

Tools and Techniques That Support This Practice

Toil tracking via tickets, time logs, or labels.
Runbook automation platforms (PagerDuty Automation, Ansible).
Self-service tooling to remove human-in-the-loop requests.
Monitoring to find and fix recurring failure causes.
Dashboards that track toil as a percentage of team time over the quarter.
Backlog labels that make toil-reduction work visible and prioritizable.

How This Practice Applies to Different Migration Types

Cloud Migration: Replace manual provisioning toil with infrastructure-as-code so environments are created and torn down repeatably.
Database Migration: Automate repetitive reconciliation and validation instead of doing it by hand.
SaaS Migration: Build self-service flows so routine vendor tasks do not generate tickets.
Codebase Migration: Automate repetitive code-transformation and verification steps.

Checklist

Toil defined and measured for the team
Largest toil sources identified
Toil ceiling target set
Highest-toil tasks automated first
Root causes eliminated where possible
Engineering time for automation protected
Toil re-measured periodically

Toil Reduction